Reversing a PAL to NTSC conversion

I wanted to capture video from an NTSC VHS tape that had been converted from a PAL tape. Of course, it would be better to capture from the original PAL tape, but I only had the NTSC tape and so I was forced to work from that.

The NTSC tape has some fields which have breaks in them. The area above the break is the next field, and the area before the break is the previous field. These fields cause a slight motion stutter, and motion is perfectly smooth without them. It's clear that they have been inserted to increase the field rate from PAL's 50 Hz to NTSC's 59.94 Hz.

Usually, the added fields occur every 6 fields. This means there are 5 original fields, 1 fake field, 5 original fields, and so on. However, the break in the field slowly moves down the field. When the break moves off the bottom of the field, then there are a few fields that are total duplicates. After that, the break moves into the top of the next field. Before that happens, it is necessary to skip ahead 7 fields once (outputting 6 original fields) to stay in sync with the inserted field. Another way to look at this is that there is a signal to go back one field which repeats at slightly more than 6 field times. It's not exactly 6 because NTSC uses 59.94 instead of 60 Hz, and so the ratio should be slightly less than 6/5.

Each inserted field flips the correspondence between PAL and NTSC odd and even fields, and in any case, PAL interlace can't really translate to NTSC interlace. (Imagine two overlapping combs with different teeth spacing.) Because of this, the converter had to deinterlace to 50 fps progressive, or more likely, just resize fields vertically while shifting them to compensate for vertical displacement due to interlace.

When outputting NTSC video, the converter bobbed the fields up and down to simulate interlace. However, this was done incorrectly. The image should bob up and down by one pixel at 480 lines or half a pixel at 240 lines, but instead, it bobbed up and down by one whole pixel at 240 lines. This finally made me give up on trying to recover interlaced video. Instead, I decided to capture at 320x480 and end up with 50 frames per second progressive video at 320x240.

The first processing step corrected for the bobbing. Using AviSynth, I added a 1 pixel border at the bottom of the top fields and cropped off 1 pixel from the top. This didn't lose any image data, because the top row in these fields was black.

Once the bobbing was corrected, I used AviSynth's RGBDifferenceFromPrevious function to collect data on inserted fields. By cropping the video so only a few lines at the top or bottom remain, I detected when the top part of the frame is the next frame or the bottom part is the previous frame. By using the function on the whole image, I detected when the switchover is in the vertical retrace interval, and frames are total duplicates. In all cases, it was necessary to crop off blackness, the very edges which are noisy, and the video head switching at the very bottom.

After a bit of experimentation, I chose to use data from frame tops, and to process it using a program which decides whether the duplicate frame occurs in 5, 6 or 7 frames from the previous one. It never occurs in 5 frames, but that capability allows the program to resynchronize if it chooses 7 when it should have chosen 6. After this, another simple program replaced the "7, 5" combinations with "6, 6" and counted the lengths of the spans of 6 that occurred before a 7.

The resulting data was good, but it had some glitches. I used a spreadsheet to work with it. There, I automatically removed some minor jitter and manually fixed a few larger glitches. When I tried to fit a line to the data, I found that it was actually a hockey stick curve with the bend at the start. This was probably because oscillators drifted during warmup and then stabilized. I considered trying to fit some kind of function to the graph, but minor fixes were sufficient.

Once I was satisfied with the data, I wrote another simple program which created a VirtualDub script. First, I had it keep the frames I wanted to remove, to ensure that I am indeed removing the frames which have tears in them. Then I created the final script which removed those frames.

After the video was finished, it was time to synchronize the audio. This can either be done by resampling the audio or changing the frame rate. I chose to resample the audio. For perfect sync, I could have used the data I generated to create a variable sample rate, but I instead just used a fixed sample rate based on a linear approximation. The errors were small enough to be unnoticeable.

Finally, it was time to encode the audio and video. I used a low (high quality) CRF in x264, because at 320x240 the video was quite sharp and I didn't want to degrade it.

Authorize.net makes it hard to get a refund after a credit card expires

I'm having difficulties obtaining a refund after the credit card expired. The merchant is willing to provide a refund, but Authorize.net doesn't allow it. After a conference call with the merchant and Authorize.net support, it turns out the merchant needs to add Expanded Credit Capabilities to their account. That service is free, but it involves filling out a form and that makes the merchant a bit reluctant.

This is happening even though the transaction was less than a month ago, the same credit card account is still active, and the new card has the same number. The merchant tried just updating the expiration date, but that didn't work.

Manually downloading Yahoo Messenger

If you "download Yahoo Messenger" from Yahoo, you actually download a small executable which can download Messenger and other things. Yahoo doesn't provide you with links to the actual executable which installs Messenger. You can get links from an .ini file you can download:

http://xp.yimg.com/gj/msgr/10/ini/ymsgr10_us.ini

For other countries, change the _us to another country code, such as _ca for Canada. The Messenger install file is located at the URL which is the INST_PATH value in the YMSGR section. Currently Yahoo Messenger is located at:

http://xh.yimg.com/gj/msgr/10/client/ymsgr1000_1270_us.exe

Copying Windows to a new hard drive with dd

Based on the existence of disk cloning software, one might assume that specialized software is needed to move Windows to a new hard drive. However, that's not true; the raw data can simply be copied. My normal personal setup has a small OS partition and a large data partition. I did the copying from Linux (SystemRescueCd) using dd. Here's the procedure I followed, with sda being the new drive and sdb being the old one (don't mix them up; that could cause data loss):

1. Copy the start sectors, before the first partition: "dd if=/dev/sdb of=/dev/sda bs=512 count=63". I could have just copied the MBR with "count=1".
2. Run fdisk and delete the data partition from the new drive. As a side effect, this causes Linux to recognize the newly written partition table.
3. Copy the system partition to the new drive: "dd if=/dev/sdb1 of=/dev/sda1 bs=1M".
4. Run fdisk and change the ID of the old drive. This is one of the advanced functions. There's a bug in fdisk: it doesn't save the partition table after this. To work around the bug, make another change. I set the type of the old system partition to NTFS hidden.
5. Reboot into Windows. It should boot off the new drive and mount the partition there as the system partition.
6. In Windows Disk Management, extend the new partition. Windows can always use more space, and new drives are usually bigger.
7. Still in Disk Management, create the new data partition.
8. Copy data to the new data partition.
9. Remove the old hard drive. It is no longer needed.
   

If there's no need to resize the system partition and the new drive is the same size or bigger than the old drive, the entire drive can be copied, simply using "dd if=/dev/sdb of=/dev/sda bs=1M".

When working with modern large hard drives and not moving partitions relative to the start of the disk, the NT boot sector doesn't need to be altered. If moving a partition (for example, to align it for Advanced Format on the new drive) it would be necessary to change "Number of Hidden Sectors" at offset 0x1C in the boot sector.

If the new drive fails, and the old drive has to be used again, the procedure is simple: just change the disk ID to its former ID and unhide the system partition.

Medieval CUE Splitter 1.2 is broken!

Medieval CUE Splitter is a seemingly nice and user friendly Windows application which can split many audio formats using CUE files. It supports many audio formats and preserves tag information. In terms of features, it seems so great, but it's broken and it loses audio data!

I already knew that splitting of MP3 files is broken. I wrote about it in the past, with version 1.0. A quick check in version 1.2 shows that it still doesn't work correctly.

Recently, I learned that FLAC splitting is also broken. The total number of samples in the output files is less than the number of samples in the input file. This means data was lost. There is also another minor bug which may actually be helpful: the MD5 value is not written to the file. This can be used to identify FLAC files may have been split using Medieval CUE Splitter, because "flac -t" will complain about it.

So far, it seems that Monkey's Audio (APE) files split without any data loss. The number of samples is correct, and the concatenated audio data from the split files matches the audio data from the source file. This means that as a workaround, other lossless files can be converted to APE and split. However, this workaround certainly slows things down, and I can't really trust the program because of the other bugs.

Don't upgrade to PowerChute Personal Edition 3.0

If you're running Windows 7 or Vista and you have UAC enabled, don't upgrade to PowerChute Personal Edition 3.0. APC doesn't know how to properly program for Windows 7 and Vista. Their system tray application needs administrative rights, and so a UAC prompt will appear after every bootup or login. APC knows about this issue, and they suggest reducing your security settings as a workaround.

Administrative rights are requested by the executable's manifest. The manifest could be edited or even removed. However, that would probably cause problems. Just use an older version of PowerChute Personal Edition; it works fine.

Prolific PL-2303 driver 2.0.2.1 causes crashes

Version 2.0.2.1 of the Prolific PL-2303 USB serial port driver is buggy. It caused an IRQL_NOT_LESS_OR_EQUAL (0x1000000a) crash twice, both times when I was sending a file to a serial port via copy in cmd and I aborted the copy with Control-C. Windows fails to find an updated driver, but a much newer driver can be downloaded from Prolific. I'm now using version 3.3.11.152, which seems stable so far. This is in 32 bit Windows 7.

Setting the MAC address on an Inspiron 6400

Dell Wireless 1500 802.11n WLAN Mini-card

The card is actually based on a Broadcom BCM4328 chip, and the drivers are from Broadcom. I'm currently using driver version 5.60.48.35. The advanced tab of properties in Device Manager allows a "Locally Administered MAC Address" to be assigned. It is an 8 byte (or 12 character) hexadecimal string using uppercase letters. It must be a locally administered unicast address, meaning the least significant bits of the first byte are 10 (and therefore the second character is 2, 6, A or E). An address which doesn't follow this format will be accepted without any error messages, but the driver will ignore it, and the card will continue to use its original address.

I wonder if the use of a locally administered address is being enforced to prevent copying of other addresses and circumvention of MAC address filtering on wireless networks. I also wonder if it would be easy to reverse-engineer the driver and remove this limitation. However, I currently have no need for this so I will not attempt it.

Broadcom 440x 10/100 Integrated Controller

I'm currently using driver version 4.60.0.1. There was no option for changing the MAC address in the advanced tab of properties in Device Manager. However, the address can be changed via the registry. The registry entry needs to be located within the adapter's numbered key in "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}", the same key where "DriverDesc" is "Broadcom 440x 10/100 Integrated Controller". Its name needs to be "NetworkAddress" and it must be of REG_SZ (string) type. The address must be entered as a hexadecimal number, with uppercase letters and no separators between bytes. After changing this entry, the adapter has to be disabled and re-enabled for it to take effect. (When changing the address via the properties dialog, Windows does this automatically.)

It's possible to add a "Locally Administered MAC Address" item to the advanced properties tab. All the items there are described by keys in the registry in the "Ndi\params" key within the adapter's key. Since the wireless card sets the MAC address in the same way, you can export the wireless card's "NetworkAddress" key, change the adapter number in the .reg file to the 440x, and then import the .reg file. Alternatively, use the following, also making sure that the number ("0007" here) matches the number of the 440x adapter's key:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}\0007\Ndi\params\NetworkAddress]
"ParamDesc"="Locally Administered MAC Address"
"type"="edit"
"LimitText"="12"
"UpperCase"="1"
"default"=""
"optional"="1"

The 440x only demands that you use a unicast address, meaning the least significant bit of the first byte is 0 (and therefore the second character is even). If the bit is not zero, the driver will ignore the address and continue using the original address. Unlike with the wireless card, the locally administered bit does not need to be set. This means the mac address of another device can be cloned or spoofed.