Sunday, November 13, 2011

Weird APC Back-UPS ES 650 BE650R-CN overvoltage behaviour

Today there is a sustained overvoltage at around 130V. I'm told they paralleled a bunch of feeder lines because of work being done today. I found out about this because an APC Back-UPS ES 650 (BE650R-CN) malfunctioned, and I noticed the "decreasing voltage" light on the line-interactive UPS which I used as the replacement.

I was told that the BE650R-CN was beeping a lot in the morning and that it stopped supplying power. When I plugged in and turned on the UPS without any load or computer connection, it initially seemed to start normally. However, when the power on self test was supposed to finish and the UPS was supposed to switch back to external power, it instead started repeatedly clicking as if it was switching back and forth between external and UPS power.

After seeing this, I took out the battery and tested it. The battery seemed charged, at over 13V, and it was capable of powering a car headlight. Later, when I put it back in the UPS, I could power a 100W incandescent lamp. The on/off button sometimes sticks and fails to pop up, but other than that, the UPS seems fine. It supplies external power, even though the external voltage is still 130V. The upper limit is set to 139V.

I'm left wondering what happened. Did I run into a firmware bug, which was fixed by removing the battery for a while? Could the sticking power switch cause this? Is the upper voltage limit stored in non-volatile memory, or was it possibly lower before and reset by removing the battery?

Building Rockbox simulator in Windows using Cygwin

When building the Rockbox simulator under Cygwin in Windows, an important thing to keep in mind is that the simulator is normally being cross compiled. It is being built within Cygwin using Cygwin tools, but it's built to run outside Cygwin, as a normal Windows application.

When using the sdl package from http://download.rockbox.org/cygwin/, tools/configure doesn't need to be told about the cross-compiling because sdl-config adds the -mno-cygwin switch. That switch tells the old GCC 3.4.4 compiler to build a standard Windows application instead of a Cygwin application. The switch is only really meant for compiling parts of Cygwin which must not depend on Cygwin, but it works.

If you instead use SDL from libsdl.org, the -mno-cygwin flag won't be added and an attempt will be made to compile the simulator as a Cygwin application. This fails with many errors in /usr/include/sys/reent.h because Rockbox firmware/include/_ansi.h is included instead of /usr/include/_ansi.h.

If you try to use the -mno-cygwin flag in Cygwin gcc-4, you'll be told "The -mno-cygwin flag has been removed; use a mingw-targeted cross-compiler."

The MinGW-w64 project provides good up-to-date MinGW compilers. Cygwin packages are available from within Cygwin setup. It's possible to use that compiler and SDL from libsdl.org by setting CROSS_COMPILE=i686-w64-mingw32- when running configure.

With either the old Cygwin compiler or i686-w64-mingw32-gcc, there are linking problems with multiple definitions. Those can be ignored by adding -Wl,--allow-multiple-definition to GLOBAL_LDOPTS in Makefile.

Friday, November 04, 2011

My VFD display project


In 2004, I built this project around a vacuum fluorescent display (VFD). I bought the VFD at American Science & Surplus when visiting a friend in Chicago. It is clearly designed for a piece of consumer audio equipment with tape and CD players. There are various audio-related indicators at the top, a 12 character 14 segment display in the centre, and two bar graphs and a two digit 7 segment display at the bottom. I use it to display Winamp status, CPU load, and a few other things. Here's a closeup of the display:

A VFD is basically an array of triode vacuum tubes. Thin and almost invisible wires stretched across the display form the cathode. They emit electrons when heated. The phosphor coated display segments are anodes. When they're significantly more positive than the cathode, they attract electrons which make the segment glow. To reduce the number of pins, segments are connected together and the display is multiplexed via the grids. To light up a segment, a positive voltage needs to be applied to both the corresponding anode and corresponding grid.

Special purpose chips are typically used to drive VFDs, because required grid and anode voltages are higher than logic voltages. I chose the Allego A6812. The chip contains drivers, a shift register for loading contents serially, and latches so that output can continue while new data is loaded. The cathode also requires its own driver circuit. Because the cathode is also the heater, there is a significant voltage drop across it. If it was powered by DC, one side of the display would be brighter than the other, and so it needs to be powered by AC. I built the driver using a 555 timer and an H-bridge. Here is the driver circuit board before the display was placed on top:

I first tried out the display using an MC68705P3S which had a small bootloader for loading code into RAM via the parallel port. This was convenient because normally, code needs to be written into a 2716 EPROM and copied to the MC68705P3S, and then the microcontroller and EPROM need to be erased via ultraviolet light before the process is repeated. After this experiment, I started working on the permanent firmware, basing it on the firmware from my LED sign. Here's a photo with the unused initial circuit on the breadboard, and the start of the final circuit on perfboard:

I had a lot of TTL chips which I easily removed from a large ISA card, and I ended up adding several of those plus some other chips to the project. I added various capabilities: 6 digital outputs, 4 analog inputs (via an ADC0804 with multiplexing via a 4016), and a shift register to accelerate loading of data into the sign. (Many modern microcontrollers have similar functionality integrated into the microcontroller chip itself.) Here is a closeup of the almost completed main board:


One of the last things I added was a simple step up converter, so the VFD anode voltage is derived from the 5V supply that powers everything else:

Like with my LED sign, the case consists of quarter inch plywood. This time, I made the case by myself. Drilling edgewise into plywood isn't hard if done slowly and carefully. Here are a few photos of the case, progressing from the two boards connected together to the final painted case:

The firmware started out with the software serial code and command interpreter from my LED sign. Thanks to various optimizations, serial communication is at 1200 baud. This time, the command prompt offers only basic functionality, because I focused a lot more on functionality that's meant to be used from software running on a computer. From the command prompt, software can enter into what I call "byte mode", where commands are efficiently encoded into bytes, and they can be sent without any waiting. In that mode, the sign can multitask, scrolling a message without interruptions, while accepting other commands. That mode is also optimized for sending bar graph values and changing numbers at individual locations in the display.

The last thing I added was standalone ability to function as a clock, with multiple "alarms" changing the parallel outputs. The crystal oscillator provides good accuracy, and it was easy to calibrate the clock based on drift observed overnight.

The software controlling the sign currently runs as a Winamp DSP plugin. That is because the main function of the sign is display of Winamp status. It's not something I really need, but I really like it. When Winamp isn't playing, the bar graphs display CPU activity. This part is really useful if I turn off the monitor for some long operation, or to monitor CPU usage when running full screen software. It also helps spot anomalous activity, such as an application taking up 100% of one core.

I've used the digital outputs for various purposes. At first, I used the clock and alarm functionality to turn on my computer when it's time to wake up. Later, I used it to upload code to MC68705P3S microcontrollers for various experiments. Most recently, I used it when first trying it the TLC5940 for my RGB lamp. I didn't do as much with the analog inputs. One is connected to a photocell, and another to a thermistor, but I don't really need that data.

You can download the firmware from Dropbox and see a few more photos in the album.

Thursday, November 03, 2011

My LED sign

I made an LED sign about 14 years ago. It consists of a 72×7 array of 5mm red LEDs. The LEDs are flangeless, which allows tight packing on standard perfboard.The array was easy to build: I just bent the pins over and soldered them together, adding electrical tape to isolate between rows and columns.
The case consists of quarter inch plywood with an acrylic front. My dad helped me with that, because it involved some tricky drilling lengthwise through plywood. I designed and built all the circuitry inside. Here's the sign with the front removed:
You can see spacers made from bits of hardwood flooring, and a bit of circuitry. The circuitry drives rows via transistors which have low saturation voltage. Here's a closeup:
The circuit boards simply slide out of the front. They consist of one board with the LED array and row circuitry, and another board with the microcontroller and column circuitry. The components face outward, and the wiring is in the centre. Here is the other side of the sandwich and the empty case:
The microcontroller controls the sign via a chain of 74HC374 octal D flip-flop chips. It outputs 8 bits, pulses the clock to latch data in the first chip and move data to the next chip, and repeats the process. While this is going on, the outputs of the last chip in the chain are disabled, which blanks the sign via the row drivers. To provide sufficient current for the rows, 7407 buffers are used. Current limiting is via the yellow DIP resistor arrays. Here is a block diagram:
The microcontroller is a Motorola MC68705P3S, from the 6805 family. I chose it because I had the chips, programmer and toolchain from a previous project. It is a very basic 8 bit NMOS microcontroller with 112 bytes of RAM and 1668 bytes of EPROM. Unlike modern microcontrollers such as the MSP430 family, it has very few registers: the accumulator, index register, program counter, stack pointer, and condition code register. Also, output levels are similar to TTL, so the first 74374 in the chain is a 74HCT374. Here's a closeup of the area around the microcontroller:

The sign works via a serial interface at 600 baud. That may seem ridiculously slow, but it's the perfect speed for receiving new columns for horizontal scrolling. A slow speed is also required due to the microcontroller's limitations. The external oscillator is 4 MHz, but 4 cycles correspond to one instruction cycle, and all instructions take several cycles. There is no UART or DMA hardware on the chip, and everything is done via a timer interrupt and assembler code. The timer interrupt runs at three times the baud rate, so when a start bit is detected, subsequent bits can be sampled in the middle third of the bit. The electrical interface to the serial port is via a MAX232 chip at the other end of the board.

Firmware functionality is quite basic. There is a command line interface with a few functions for direct human use and a few functions which work with raw binary data. The human interface consists of an interactive drawing mode, printing of text via the on-board character generator, sign clearing, lamp test, and returning the sign contents in human-readable form via the serial interface. Due to limited EPROM space, the on-board character generator just supports one font with uppercase letters and some symbols. Other functions support scrolling up and to the left, uploading arbitrary bytes, and uploading groups of columns. These functions allow uploading of raw sign contents, where a character generator running on a computer can provide additional flexibility.

On various occasions I spent time working on PC software for the sign. I first built an interface library. Then I built a server which accepts requests from clients, along with a library that clients use to connect to the server. This allowed multiple programs to use the sign at the same time. Finally, I added support for messages which appear at most and/or at least for a specified time. This should allow the sign to be used for notifications. However, I never used it for a sustained period, because such a large notification display isn't very practical.

Besides looping a short message, there is no support for standalone operation. That means the sign is not able to display various messages with effects one would see on a LED sign that's used for advertising. I don't regret this lack of functionality, because I don't think I'd find that useful. However, if I was building something like this now, I would definitely use a more powerful microcontroller which could support that and more. I thought of replacing the MC68705P3S, but that only makes sense if I have some usage in mind.

You can download the firmware from Dropbox if you're interested. I developed it using the Motorola Portable Cross Assembler (PASM) version 1.0 toolchain.

Sunday, October 30, 2011

My MSP430 based RGB light

The Design

For a long time I thought it would be nice to have a customizable colour light, but I didn't get around to building it. This year, inspired by the MSP430 LaunchPad and my purchase of a TLC5940 with SparkFun Free Day funds I finally built it.

For the LED, I chose the 10W 500-Lumen Multi-Color RGB LED Emitter Metal Plate (140 degree) from DealExtreme. It's a bright LED at a good price. Similar LEDs are available from multiple Chinese sites. I chose DealExtreme because I like the way their site is organized, they have a good reputation and good prices.

Initially, I was disappointed with the TLC5940. Yes, it is a "16 channel PWM unit with 12 bit duty cycle control", as described on SparkFun. However, it requires an external clock and PWM cycle start signal, with some very specific timings when loading new PWM data and starting a new cycle. Generating all that precisely would require a lot of resources from the microcontroller.

My first circuit clocked the TLC5940 using a 555 timer, and restarted the PWM cycle using a simple R-C network. The PWM data was bit-banged from my computer, and not in any way synchronized with the PWM cycle. This worked, with the main disadvantage being that the method was slow and not suited to colour changing effects.

I spent quite a bit of time wondering how to satisfy the timings in the datasheet while having low CPU utilization and fast PWM speed. My first design used a 7474 dual D-type flip-flop. A flip-flop in toggle configuration generated SMCLK/2, and I used the CLR input to extend the GSCLK cycle at the end of the PWM cycle.

This worked well, but I ended up choosing a simpler design, connecting SMCLK to GSCLK without glue logic. BLANK was generated from TACCR0, with the timer in up mode, and output mode 3. This creates a 1 cycle pulse, using only one compare register. This does not guarantee the proper timing between the rising edges of GSCLK and BLANK, but it works perfectly. I feel it's okay because this is just a personal project, and because that GSCLK edge is after the 4096th edge which ends the PWM cycle.

For XLAT, the nicest solution would be to use the multiple TA0.0 outputs, and enable XLAT via P1SEL. However, I chose to use USI in SPI mode to load PWM data more quickly, and so the other TA0.0 output wasn't available. I instead connected another pin to XLAT, with a 1 kΩ resistor between BLANK and XLAT. When the XLAT pin outputs a low, XLAT is inhibited, and when it is an input, XLAT is pulsed when BLANK is pulsed. It's unfortunate that even the 20-pin MSP430 Value Line chips cram most special functions into the 14-pin footprint.

I wanted the light to have both computer control and a user interface. I chose a serial port for computer control. A capture/compare register can be used to build a nice software UART which is not affected by interrupt jitter. I based my code on msp430g2xx1_ta_uart9600.c from the TI sample code. I wasn't too happy with the DCO tolerances however. They are sufficient for serial communication if the other side is precise, but I wanted something that would use up well under half the error budget. (Maxim AN2141 (PDF) provides a nice explanation on the subject.) The MSP430 Value Line chips don't support high frequency crystals, and according to the datasheet, they can't even accept a high frequency external clock input. It is possible to provide a high frequency external clock, but I didn't want to rely on this undocumented feature, so I used a watch crystal. The crystal triggers the watchdog timer interrupt 4 times a second, and code calculates the length of one bit in SMCLK cycles, based on the crystal. To avoid PWM jitter, I don't actually change DCO settings like an FLL, and I set up the DCO without modulation. I chose a high frequency, just under 16 MHz, so the PWM rate is high and calculations are finished quickly. Serial communication is at 9600 baud, which allows colour changes up to about 192 times a second, with the three 12 bit values packed into 5 bytes.

With all the pins needed, it became difficult to use a 14-pin chip. Some tricks and compromises could have allowed it, but I didn't really like those ideas. I got an MSP430G2252 in a 20-pin package.

For the user interface, I used two switches and three potentiometers. One switch selects between off, serial control and local control, while the other selects between RGB, effects and HSV during local control. I had thoughts of using the comparator to measure pots, but I went for the easy solution, using the ADC10. I was disappointed at the noise, even with proper bypass capacitors. To mitigate the issue, code performs smoothing. Rotary encoders would be a better alternative, but I have plenty of pots, and quadrature encoders would need more pins. With a rotary encoder, it would be possible to avoid colour changes when changing between RGB and HSV input modes, and instead just allow further tweaking in the new mode.

The key software component of the local user interface is a multiplication routine, which multiplies two 16-bit values as fixed point numbers between 0 and 1. That same routine is used for gamma correction, HSV to RGB conversion and fading. For gamma correction, values are simply squared. A power of 2.2 might be more accurate, but squaring is close enough. HSV to RGB conversion is done via a highly optimized assembler routine, partly just because I had fun writing it. Fading via Bresenham's line algorithm would have been more efficient, but the multiplication based version was fast enough, and code size matters when only 2 KB of flash is available.

After all that was done, one pin and some flash space remained. I used the pin as a serial output, so the computer could read the current colour and potentiometer positions. Due to the special purpose pins being crammed into the 14 pin footprint, I wasn't able to directly output from TA0.2, and so the output is done from the interrupt handler. It's not ideal, but it works. I consider it to be a bonus feature.

The gamma correction in the local interface is necessary, but it created a problem. Fading requires linearly changing the value before gamma correction, but the original serial interface only allowed setting the raw PWM value, which is the result of gamma correction. For proper fading, the code would require the corresponding value before gamma correction. The serial output can help here, by allowing the current setting to be read and initial fading to be done on the computer. The serial input also allows input of values before gamma correction. For a proper fade when switching away from serial mode, code can either use that all the time or just use it once before quitting.

Originally, I had various ideas for colour changing effects. Due to the limited code size, I ended up only implementing hue spinning, with the ability to set brightness, saturation and speed. The speed selection allows a wide range, from rotations taking several minutes to such rapid rotation that the light seems steady but fast movement leaves coloured trails.

The circuit is relatively uninteresting. Mostly, it's a matter of point to point connections between chips. I used an LM317 to supply power to the MSP430 and TLC5940. At the inputs, single transistor inverters perform level shifting and provide some protection for the MSP430.

The biggest difficulty with the circuit was the TLC5940's power dissipation. With a 10W LED and 2456mW maximum power dissipation, I had to be careful to avoid overheating the chip. The light requires a 12V regulated wall wart, and I added resistors to limit the voltage drop at the chip to about 1.4V. I also mounted the chip on the underside of the circuit board and connected it to the metal bottom panel using thermally conductive putty.

According to calculations, the chip isn't close to its limit, but it's nice to have a big safety margin. The microcontroller also monitors the XERR pin via an interrupt and turns off the light if the TLC5940 overheats. I'm satisfied with the power losses due to the linear current regulation. However, considering the power losses and resultant heat dissipation, if driving a higher power LED or array I would choose three switch-mode LED drivers instead of the TLC5940.

Usage experiences

In the local interface, I mostly use the HSV input mode. It's far more convenient than the RGB mode. HSV is kind of stupid, because it ignores many perceptual factors. (The three primaries have different apparent brightness. When multiple primaries produce a colour, there is an increase in brightness and decrease in saturation. Colour does not seem to vary at a fixed speed as H is changed.) However, more complex colour spaces such as CIE LCH have many colours which cannot be reproduced via the three primaries. If the three potentiometers set L, C and H, there would be settings that are out of range. I tried it via the serial interface, and it was quite confusing. HSV to RGB conversion is also a much simpler algorithm and more suited to small microcontrollers. I have no regrets about choosing HSV.

So far, my favourite computer-based effect is a Winamp plugin which divides the spectrum into 3 zones, and sets red, green and blue values based on an exponential moving average of sound intensity in the corresponding zone. I used red for the lowest band and blue for the highest band. It's very nice with some types of music.

The Code

I'm releasing the firmware under the GNU General Public License (GPL) version 3, because I like how the GPL encourages creation of free software. I developed and compiled the code using IAR Embedded Workbench KickStart, because it offers a nice IDE for developing and debugging. Pin assignments are listed in the header file. If you want to change them, consider that many of the port 1 connections depend on the special functionality available on certain pins. You can download MSPRGB source code from Dropbox.

I'm separately releasing some code which uses the RGB lamp via a serial connection. The zip you can download from Dropbox contains librgb, a library for interfacing to the lamp, and vis_rgb, the Winamp plugin I described. I just cleaned up librgb and improved portability. I'm not protecting librgb and vis_rgb via the GPL because I don't feel it is especially worthy and because I don't want to restrict its usage.

Synaesthesia 2.4 for Windows

It seems like a lot of "music visualization" plugins just use music as a random seed for interesting visual effects. There is a link between changes in music and changes on screen, but there's no real collection between individual elements of music and individual elements on screen. The best software I know of for visualizing music is Synaesthesia. It presents an image with the horizontal axis being stereo and the vertical axis being frequency. Images can directly correspond to individual sounds, and so I can see sounds on the screen as I'm hearing them. The program truly deserves the name Synaesthesia.

Until now, the only publicly released Windows version was an old port based on Synaesthesia 1.2. The current version of Synaesthesia is 2.4, and there are many new features since 1.2. A long time ago I started working on a port of Synaesthesia as a Winamp plugin. However, I learned that GPL licensed code (Synaesthesia) can't be used in a plugin for an application that has an incompatible license (Winamp). I just created a new port based on Synaesthesia 2.4, the 1.2 Windows port and some of my code. You can download it from Dropbox. Further information is available within "Synaesthesia 2.4 Windows port readme.txt" in that zip file.

This is just a quick port I created today. There may be bugs. Please don't bother the original authors of Synaesthesia and the 1.2 port. If you run into problems, leave a comment here.

Friday, October 28, 2011

Winamp plugin which enables UAC virtualization

Some old Winamp plugins attempt to store settings in the plugin directory. This fails in Vista and Windows 7 because directories under Program Files aren't writable for ordinary users. Windows has a feature, UAC Virtualization, which redirects these writes to a folder in the user's profile (%USERPROFILE%\AppData\Local\VirtualStore). However, the manifest in winamp.exe disables this feature.

It's possible to grant write access to the directory, toggle UAC state via Task Manager, and edit the manifest in winamp.exe. However, all of these workarounds have disadvantages. Because of that, I just created a simple plugin which enables UAC. It may not execute early enough to help code which runs when Winamp starts, but it's perfect for visualization plugins which run on demand later. You can download it from Dropbox. I'm not releasing the full source because most of it is just sample code from the Winamp SDK. Enabling UAC Virtualization is easy:

HANDLE token;

if (OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_DEFAULT, &token)) {
    DWORD v = 1;
    SetTokenInformation(token, TokenVirtualizationEnabled, &v, sizeof(v));
    CloseHandle(token);

}

Thursday, October 27, 2011

The PL-2303 code 10 error

Some cheap PL-2303 based USB to serial adapters don't work with new drivers from Prolific. In Windows, Device Manager gives a "This device cannot start. (Code 10)" error. Prolific knows about this error and has a FAQ question about it. To me, the answer there seems to imply that these PL-2303 chips may be counterfeit.

When I ended up with such a bad PL-2303 based  adapter, I investigated the issue using Snoopy Pro. It is a free open source (GPL) program which logs USB communication between the driver and device. SnoopyPro allowed me to see the requests sent by the Prolific driver, and compare responses.

Here is the request sent from Prolific v1417 driver in 32 bit Windows 7 SP1, with version 3.3.17.203 ser2pl.sys:

URB Header (length: 80)
SequenceNumber: 10
Function: 0008 (CONTROL_TRANSFER)
PipeHandle: 00000000

SetupPacket:
0000: c0 01 86 86 00 00 02 00 
bmRequestType: c0
  DIR: Device-To-Host
  TYPE: Vendor
  RECIPIENT: Device
bRequest: 01  

No TransferBuffer

Here is the response from a Dynex DX-UBDB9 adapter, which works perfectly:

URB Header (length: 80)
SequenceNumber: 10
Function: 0008 (CONTROL_TRANSFER)
PipeHandle: a24f2724

SetupPacket:
0000: c0 01 86 86 00 00 02 00 
bmRequestType: c0
  DIR: Device-To-Host
  TYPE: Vendor
  RECIPIENT: Device
bRequest: 01 

TransferBuffer: 0x00000001 (1) length
0000: aa

Finally, here's the response from the BAFO USB to RS232 Converter Adapter that I got from DinoDirect:

URB Header (length: 80)
SequenceNumber: 10
Function: 0008 (CONTROL_TRANSFER)
PipeHandle: 856e8594

SetupPacket:
0000: c0 01 86 86 00 00 02 00 
bmRequestType: c0
  DIR: Device-To-Host
  TYPE: Vendor
  RECIPIENT: Device
bRequest: 01

TransferBuffer: 0x00000001 (1) length
0000: 00

There's one difference: the good adapter responds with 0xAA and the bad adapter responds with zero. The information could be used to find the check in ser2pl.sys and make the driver accept the zero. However, the EULA doesn't allow that. In any case, the adapter from DinoDirect has other problems which cannot be fixed.

Why you need to be careful when buying a USB serial adapter

I got a BAFO USB to RS232 Converter Adapter from DinoDirect. I thought I was getting a good deal. It was more expensive than the no-name adapters available elsewhere, but BAFO is an actual brand with support and drivers. I noticed that they were distributing a relatively recent Prolific PL-2303 drivers, which should mean they used PL-2303 chips which didn't trigger the code 10 error. What I got was a piece of junk:
  • Recent drivers fail to work with the adapter, giving a code 10 error. Even the most recent drivers downloaded from BAFO fail. (I wonder if it is really a BAFO product.)
  • The output is 0-5V. The adapter cannot output negative voltages like DinoDirect claims. (They write "The output voltage of our USB to RS232 converters (Pin 3: TX line) is about -9VDC".)
  • The plug is loose, and the USB connection is disrupted if the adapter is moved.
  • In Linux or in OS X with the open source drivers, the adapter starts endlessly repeating the last character that is received. (Prolific's drivers don't work in OS X either.)
  • The adapter cannot detect RS-232 break.
  • Timing may be inaccurate at higher baud rates
The adapter is usable for some applications in Windows with older drivers. However, watch out: some old drivers from Prolific cause bluescreens.

When I complained to DinoDirect, I was told I'd get a $5 giftcard with which I could "pay as cash" , but I was given a $5 off $20 coupon. When I tried to post a review listing the problems the review didn't get approved. A positive review for another item was approved immediately, and I got DinoPoints. When a second attempt to post the review didn't get approved either, I decided to write this blog post.

I should have done some research before dealing with DinoDirect. It seems like a lot of people dislike them. Some even feel that DinoDirect is a scam. I think it's pretty clear they're not a scam, but I can't trust the information on their website, or that they will respond to my problems in a satisfying way. (DealExtreme would probably give store credit for a product like this.) After browsing the DinoDirect web site more, I found a lot of questionable reviews and even questionable product information. For example, some macro diopters have information and reviews as if they're polarizers. Overall, many reviews seem questionable. I'm not saying that DinoDirect is posting fake reviews. Maybe some users post lots of reviews so they can get DinoPoints and free stuff. DealExtreme also offers points for reviews, but their reviews seem genuine. Go ahead and compare DealExtreme and DinoDirect reviews. Also, note that DealExtreme has a lot of activity in the per-product forums.

DinoDirect isn't all bad. They gave me a $1 gift card, I guess for creating my account. Then, I got 100 DinoPoints, corresponding to $1, for a short survey. When the flashlight I ordered wasn't in stock, live support they gave me a $3 gift card. Finally, I ordered the RICHONG RC-7001 LED Flashlight (1*AAA Battery) flashlight and received it. It's a decent little LED flashlight. It seems pretty rugged, and I like how it uses one AAA battery instead of several coin cells. Getting that for free partly makes up for the USB to serial junk I paid for.

Finally, if you really want to order from DinoDirect, two suggestions: Try to find some better information on the product elsewhere. Check for coupon codes, because DinoDirect is always running promotions. RetailMeNot has a good list.

Monday, October 24, 2011

The Unix serial port output buffer

Unix serial ports have associated input and output buffers in the kernel. When writing data to the port, if sufficient space is available in the output buffer, the write call returns after copying the data into the buffer. The hardware slowly outputs data from the buffer, freeing space for more data. A program can easily fill up the buffer, because the CPU works much faster than the serial port. When insufficient space is available in the buffer, a normal blocking write will wait. It will not simply wait for sufficient space; instead, it will wait for the buffer to empty to a specific low limit.

When simply sending large amounts of data, this behaviour is appropriate and efficient. Programs don't need to constantly wake to feed more data to the hardware. Instead, they can put a lot of data into the buffer and then sleep until the buffer gets to that low limit. However, for some applications the buffer is a problem.

Bytes that are already in the buffer form a delay between when a new byte is written and when it is actually output from the port. (All those bytes need to be output first.) As a result, serial responses and actions from serially controlled hardware may be delayed. If a program performs checks to see whether to continue sending data, that will also be delayed when a write call waits for the output buffer to empty.

In Linux the output buffer is one page (typically 4KB), and the process is woken when only 256 bytes (set via WAKEUP_CHARS) remain. That may seem like a small amount of memory, but at low baud rates, it corresponds to a long time. There is no standard way to change the buffer, and in Linux, it could only be changed by editing kernel source code and recompiling the kernel.

It is simplest to wait for data to be output after writes. This can be done by waiting for responses after writs or via tcdrain. (Other methods such as fsync and O_SYNC cannot be relied on for serial ports.) Unfortunately, this defeats the efficiency benefits of buffering.

When the issue is a need to stop output quickly, instead of a need for constant low latency, data in the buffer can be discarded via tcflush. However, this gets tricky. Since writes can wait for a long time, some form of asynchronous I/O is needed. Once the buffer is flushed, serial controlled hardware may be in an unknown state, which complicates things.

As an added benefit, tcflush and tcdrain also deal with the hardware buffer. In most built in serial ports, these buffers are tiny. However, USB to serial adapters can have large buffers.

Friday, October 14, 2011

In Linux, use the Realtek r8168 driver for RTL8168

I have a Gigabyte GA-P35-DS3R motherboard with a RTL8168 Ethernet chip.
When using the r8169 driver that's part of the Linux kernel, I often lose the connection after S3 suspend. Usually, taking the interface down and back up with ifconfig fixes it, but unloading and reloading the module is sometimes necessary. This problem has been around for a while, and it still exists in the current Linux Mint Debian Edition 3.0.0-1-amd64 kernel. Even adding r8168 to SUSPEND_MODULES doesn't help.

The best solution seems to be using the r8168 driver from Realtek instead. It solved the problems after suspend, and seems to have also sped up suspend a bit. It may have also fixed some hangs I used to get occasionally.

At first, version 8.025.00 refused to build because the src/Makefile didn't recognize that Linux 3 should be treated like Linux 2.6. I fixed this by changing the lines setting KEXT and KFLAG to:
KEXT := ko
KFLAG := 2x
After this, I had no further problems and autorun.sh correctly built and installed the driver. It renames r8169.ko to r8169.bak to deactivate that driver. After confirming that r8168 works properly, it's a good idea to blacklist r8169. It's also necessary to update the initial ramdisk, because r8169 may be loaded from there.


Friday, September 23, 2011

SkyDrive: 25 GB of free storage that's too annoying to actually use

For several years, I've known that I have a Windows Live SkyDrive account with 25 GB of free storage. However, I never actually tried to use it. Today I tried to use it and I see what's the catch: it's too annoying for storing large amounts of data.

I was hoping to upload all of my photos. Theoretically, it seemed like a good idea. The photo viewer is great, the per-file size limit is not an issue, and the overall size is more than other free services allow but only a fraction of the 25 GB limit.

The first problem is that the web uploader is quite limited. It can't upload folders, so I would have to manually create them and upload their contents. It doesn't even support drag and drop, except for in IE. Since uploading would be a lot of work, I rejected this option.

Then I learned that SkyDrive can be accessed via WebDAV and mounted as a drive in Windows. (It uses proprietary Passport authentication, so I'm not sure if any other clients can access it.) This was easy once I sorted out some firewall issues. Unfortunately, only document file types can be uploaded. Attempts to upload JPEG images and archives result in an error telling me that the file "is too large for the destination file system". It's not a file size issue because it happens with small files. I guess Microsoft only wants the WebDAV access to be used for Office documents.

Since PDFs are supported, I thought about using using them. JPEG images can be included in PDFs without re-encoding, and PDFs also support attachments for other file types. However, this is a kludge without any benefits besides WebDAV access, and so I decided to try ordinary multi-part archives instead.

Archives can be uploaded via the web interface, and files of 104,857,600 bytes (100 MB, 100 * 2^20) may be uploaded. Unfortunately, this is unreliable. The Silverlight uploader only seems to allow one 100 MB file to start uploading in a batch, and even that file seems to stop before the end. The upload seems to complete successfully, because the downloaded file is bit-identical to what I uploaded. The classic form based uploader also fails before the end, and I'm not sure if it's usable for large files.

Some per-service firewall rules don't work in Windows 7

The Windows 7 Firewall with Advanced Security allows you to specify a service in addition to a process. This is important because many services run under svchost.exe. Rules which simply reference svchost.exe may be too broad.

This works well with  the Windows Update and Windows Time* services. When outbound connections are disabled by default, rules can be created to permit those services. However, it doesn't always work.

In the past I spent a lot of time trying to figure out how to permit Windows Media Center schedule updates. I guess that part of the update is via the Background Intelligent Transfer Service (BITS). I said "I guess" because "netstat -b" sometimes showed them using the User Profile Service (ProfSvc), which doesn't make sense. Rules which permit outbound communication from BITS, ProfSvc, and even a large number of other services never accomplished anything. I finally gave up and created a rule permitting outbound communication on port 80 by svchost.exe (without specifying a service).

I was just trying to figure out how to allow the WebDAV client through the firewall. It obviously uses the WebClient service. In this case "netstat -b" also shows this. However rules permitting outbound communication by the WebClient service do nothing. I was forced to create a rule permitting outbound communication by svchost.exe on port 443 for SSL WebDAV access.

I'm not sure if these are bugs or intentional limitations. When creating a rule for svchost.exe, I am warned that "Windows services have been restricted with rules that allow expected behavior only. Rules that specify host processes, such as svchost.exe, might not work as expected because they can conflict with Windows service-hardening rules." Maybe Microsoft chose to not permit rules involving BITS and WebClient because they are general purpose communication services which could also be used for malicious purposes.  However, it doesn't make sense that a rule specifying svchost.exe and a particular port is allowed, but when the rule is further narrowed by specifying a service, it is ignored.

* If Windows Time Service is not allowed Internet access, the first "Update now" attempt will fail with a timeout. However subsequent attempts may appear to succeed, and the time when the clock was synchronized will be updated.

Thursday, September 22, 2011

Brown polarized sunglasses are great!

I used to wonder why someone would want brown or amber sunglasses. Why add a colour cast to everything? Now I know. The colour:
  • decreases sky brightness, equalizing the overall brightness of the scene
  • removes glare, while still making the scene appear bright in a pleasant way
  • accentuates green foliage and various other colours
  • makes hazy days seem sunnier
  • helps decrease glare when driving toward the sun
(Blue-grey sunglasses are the opposite. They dim things relative to the sky, they make foliage look worse, and they aren't very effective at decreasing glare. I don't want those ever again.)

Polarization is also an important enhancement. It:
  • decreases the intensity of annoying bright reflections from non-metallic surfaces, such as reflections from the paint of cars or bodies of water
  • increases colour saturation by removing reflections (Foliage reflects a lot of light, and looks much greener when that reflection is blocked. Water surfaces also reflect light, and when that reflection is blocked, the colours of the water itself stand out.)
  • makes the sky darker and bluer in certain directions, making clouds stand out and helping equalize the overall brightness of the scene
The only downside of polarization is that the intensity of the effect depends on the orientation of the glasses.  It is most optimal when the sun is highest in the sky. Near sunset, the sunglasses need to be turned sideways to create a strong effect. Sometimes, I thought about creating electronically controlled polarized sunglasses using camera filters and a microcontroller.

When buying polarized sunglasses, it's important to check whether the effect works well. Turn the glasses and block some reflections. Cheap glasses may not be able to block reflections very much, or the blocked reflections may show a strong purple colour cast.

Monday, August 29, 2011

Stores selling cheap stuff from China

There are many online stores which sell a wide variety of cheap stuff from China, usually with free shipping. The prices can be impressively low, but the quality is sometimes questionable.

DealExtreme seems to be the most popular one. So far, I've ordered several items from there, and overall I've had a good experience. I'm using the 10W 500-Lumen Multi-Color RGB LED Emitter Metal Plate (140 degree) in my RGB lamp, and I'm happy with it. Their 5mm white LEDs are also good, with a nice colour and good brightness. I also got a nice 55mm lens cap.

So far, I've only had one problem with DealExtreme, and that was resolved in a good way. I ordered what was at the time advertised as a Massa circular polarizer for my camera. The packaging and item I received made no mention of Massa, and the item looked different. When I contacted DealExtreme, I got some store credit. The polarizer does provide decent photos as long as I don't zoom much, and I really like some photos I took using it. I also see that DealExtreme updated the item description and photos, showing what I got. As a result of all of this, I'm satisfied.

My favourite thing about DealExtreme is the reviews and forum posts. They seem genuine, and they provide useful information about the products. Yes, DealExtreme does sell some bad products, but the reviews and other information make it possible to avoid most of these.

The only real complaint I have is that some simple actions (such as redeeming store credit or DX points) require customer service interactions. These things would be easier and faster if automated. However, customer service always responded promptly, so this is not a major issue.

The main thing that motivated me to write this post is my first purchase from DinoDirect and what I saw when I spent more time on that site. It's a big contrast; DealExtreme is much better. For example, check out what other people have to say about DinoDirect, and look at the quality of item reviews on DinoDirect.


Monday, August 15, 2011

I'm still using Adobe Reader

Despite the bloat and all the people who hate it, I'm still using Adobe Reader to view PDFs in Windows. Recently, it was slow on one huge PDF on my laptop, so I decided to try out some alternatives.

Sumatra PDF was pretty good and a bit faster, but zooming wasn't as user-friendly. I didn't like Foxit Reader's user interface. It seemed like an old badly designed Linux application. PDF-XChange Viewer was pretty good, but it also had an inferior zooming user interface. The biggest problem was that all three had slow search in large PDFs, without acceleration via saved indexes like Adobe's Fast Find.

It would have been nice to switch to something leaner, but it's not really a problem. I found that disabling 2D graphics acceleration in Adobe Reader resolves the performance issues on my laptop.

Sunday, August 14, 2011

Bad capacitors and ESR testing

I have an old Soltek SL-65KVB motherboard. It was taken out of service due to AGP graphics instability which was probably due to bad capacitors. This diagnosis seemed pretty obvious because capacitors right by the AGP slot were bulging, but I didn't bother fixing the motherboard because it was a good time for an upgrade.

Now I would like to repair the motherboard, mainly because the combination of a Pentium 3, 640MB of RAM and an ISA slot could provide a nice way to use old hardware. By now, even more capacitors are obviously bad. However, only the GSC 1000µF and 1500µF 6.3V capacitors with gold/brown lettering have obvious problems and more numerous smaller GSC capacitors with silver/white lettering seem fine.

I didn't feel like unnecessarily replacing the smaller capacitors, so I quickly built a very simple capacitor ESR tester. The basic idea is the same as various other ESR measuring devices documented online: a stepdown transformer outputs low voltage and low impedance pulses which are used to measure capacitor ESR. I built the stepdown transformer from an inductor with a short secondary wrapped around it, and I used a 555 chip to supply low duty cycle pulses at high frequency. The 555 can supply enough current that it's possible to directly drive the transformer with a resistor in series with the primary.

My first attempt with a secondary of just a few turns worked pretty well when capacitors were plugged into the breadboard. However, test leads had enough impedance to make the circuit useless. Reducing the oscillation frequency to around 100 kHz and adding more turns to the secondary improved performance, making the bad capacitors very obvious. Test leads were still a problem however. Adding a 1Ω resistor in series with the secondary improved the situation significantly, so I proceeded to test the small capacitors on the motherboard via test leads.

It seems the small capacitors are all ok. However, among the large capacitors, even most that weren't bulging are bad. Only two of the ten 1000µF capacitors have a low ESR. Of course, considering the other failures, these shouldn't be used for anything either.

Wednesday, August 03, 2011

g++ dyamic linking fail

I have a DLL which works when built, but stops working later. Simply rebuilding it creates another copy which again works and stops working later. LoadLibrary returns error 998, which mean "invalid access to memory location". In other words, a crash occurs in the DLL being loaded. The crash occurs when calling _initterm from code at the DLL entry point. At __imp__initterm there should be a pointer to _initterm in msvcrt.dll, but the location instead still contains the value that's in the file on disk. As a result, if msvcrt.dll is located at a different location, the program jumps to the wrong location. I guess this happens because msvcrt.dll is imported two times. The first import just uses _write, which is properly dynamically linked. The second import lists all the other functions that are used, and dynamic linking wasn't done for any of those. The _write from the first import is just used to print "pure virtual method called\n".http://www2.blogger.com/img/blank.gif

I suspect that g++ is to blame, but I'm still not sure. The problem is now happening with g++ 4.5.3 from the 32-bit MinGW-w64 Cygwin package, but the same symptoms occurred when using the old Cygwin compiler that is based on GCC 3.

Update: The problem seems to be related to statically linking libstdc++. The duplicate msvcrt.dll exists when using -static or -static-libstdc++ (new in GCC 4.5).

Update 2: Packing the DLL with UPX 3.07 seems to fix the problem. When a packer compresses the IAT, it needs to handle dynamic linking. Apparently UPX is able to properly handle the defective IAT.

Wednesday, July 20, 2011

No, I can't trust Windows with very rare operations.

While my computer was writing to a 3½" floppy disk via the floppy controller on my motherboard, I pushed the sleep key, requesting S3 sleep. Right after that, I remembered how Windows cannot really be trusted when doing unusual things. I started to wonder if I'll see a bluescreen, but all I saw was sleep being delayed by lengthy floppy activity. When I came back, I was not too surprised to see a bluescreen. The source of the crash was a bit surprising however: usbser.sys had submitted an IRP that was already pending in the USB bus driver. I guess delays from the floppy driver triggered a bug in usbser.sys. This was failure bucket 0xFE_usbser!RestartNotifyRead+73. It was usbser.sys 6.1.7601.17514 (win7sp1_rtm.101119-1850), so this really is a component of Windows, and not a 3rd party driver.

Tuesday, May 31, 2011

Review of battery and AC adapter from LaptopCharge.com

About a year ago, I got a new LDE203X battery and AC19V90K1 AC adapter for my Inspiron 6400. I ordered from LaptopCharge.com, and I'm satisfied with that choice, the site and the items.

Basically, there were three choices for getting a battery and AC adapter: Dell, eBay, and other stores like LaptopCharge.com. Dell ridiculously overprices these things, so I didn't want to buy from them. However, I noticed that some people didn't trust third party batteries, and many people said third party batteries wear out more quickly. I also noticed that most eBay sellers didn't list a brand or part number for their batteries, which seems shady. I chose LaptopCharge.com because is a store that has been around of some time, and they sell Dr. Battery brand products. The price wasn't much higher than the best prices on eBay.

After placing my order, it arrived promptly and in good condition. Everything was packaged well inside, and the battery and AC adapter were in their own boxes. I was however unsure about whether the Dr. Battery brand meant anything. The logo only appeared on hologram stickers which were haphazardly stuck onto the battery and adapter.

The battery is a good, though not perfect fit. It takes slightly more force to engage the latches, and the thicker side sticks out a fraction of millimetre. This is totally inconsequential.

The battery charges properly and only gets very slightly warm. When it's done charging, the battery light keeps flashing occasionally, like during the topping up part of a charge. Also, if the battery is discharged to 98% or 99% and power is plugged in, Windows reports that the battery is charging but the level doesn't increase. In both of these situations, BattStat shows about 11 mW of power input, and a voltage which is well below 4.2V per cell. This seems like a very slight miscalibration of the current gauge. None of these things are a problem. (It is normal for lithium-ion batteries to require more than a few percent discharge to trigger charging.)

Just like the original battery, the replacement battery has a LED bar graph for checking battery level. It works, although the button that activates it requires more pressure and doesn't have a nice clicky feel. This is a bit of a disappointment, but certainly not a big problem.

Low battery shutdown works perfectly. The capacity decreases smoothly, and hibernation is triggered appropriately. I did not test discharge to the point of power cutoff.

During the last year, I was careful with the battery. I took it out when using my laptop on AC power for extended periods, and whenever I stored the battery, I kept it at around 40%. I still don't see any capacity degradation.

The charger looks very similar to the Dell charger, but it is bigger. This may be because it is 90W, while my Dell charger is 65W. It has a longer AC power cord, but a shorter DC cord. This is less convenient than the Dell charger. The AC cord doesn't bend where it exits the charger, which is nice, considering that I don't want to wrap the cords on the charger because of the stress that it causes.

I don't like the original Dell charger because it is noisy. It makes chaotic buzzing and beeping noises when the laptop is running and especially while the battery is charging. When in sleep mode, it produces a chirping sound as the power LED pulses. When I'm going to sleep in a quiet place, this sounds kind of like night crickets, except it's kind of annoying. The Dr. Battery charger is much quieter. I need to be closer to hear the sounds it produces, and there are no bothersome sounds in sleep mode.

Even though the Inspiron 6400 shipped with a 65W charger, it properly recognizes the new 90W charger and makes use of the additional power. With the original charger, charging slows down when the laptop is on and doing demanding tasks, and with 90W available, charging does not slow down. (BattStat can show these things.) However when charging and doing ordinary non-demanding things, the charger gets so hot that it is on the verge of causing 1st degree burns. Because of this, I wouldn't like to do constantly demanding things while charging or use the charger with a laptop which is designed to require 90W.

When I was wondering about the bar graph on the battery, I contacted Dr. Battery and LaptopCharge.com and got prompt replies. Based on this, it seems that the technical support and warranty can be trusted.

Saturday, May 14, 2011

A quick OS comparison

Windows is the de-facto standard OS. Over time, it has been enhanced to the point where it's stable and it contains practically all the features one might want. Its main advantages and disadvantages relate to its popularity. It has the greatest selection of software and support for a very wide variety of hardware, but it also has the greatest variety of malicious and deceptive software that users have to watch out for.

Mac OS X is pretty and it provides a pleasant user experience. However, it has various limitations, some of which can force one to use Windows for some tasks. It's also tied to expensive Apple computers, and it has worse hardware support. It's a good choice for those who don't do stuff that runs into the limitations and don't mind paying extra for computers.

Linux is free and there is a large selection of free software that can run on it. Some of that software is incomplete, buggy and/or ugly, but there is high quality free software. Linux has some limitations due to not being the de-facto standard, but the situation is better than with Mac OS. The main disadvantage is that a lot of GUI software is kind of awkward and not pretty, as if it was designed by programmers rather than skilled user interface designers and graphic designers.

Monday, April 18, 2011

Getting a TLC5940 to retrigger itself

At first glance, the TLC5940 seems quite nice. It offers 16 output channels, each with 12 bit PWM and a 6 bit current limit. Unfortunately, the chip doesn't have an oscillator and when one PWM cycle ends, it needs to be retriggered. This is kind of annoying and it partly defeats the purpose of using an extra chip for PWM. It means that you have to supply a jitter-free clock for PWM and you need to count cycles and retrigger the chip via BLANK when necessary.

I wanted to get it working quickly, so I devised a hack. By capacitive coupling OUT15 to BLANK, the rising edge of OUT15 at the end of the PWM cycle became a positive BLANK pulse which started the next cycle. I used a 4.7 kohm pullup at OUT15 to create a signal there, a 22 nF capacitor between the pins, and a 100 kohm pulldown at BLANK. At BLANK, I also added diodes to the power supply rails, to clip the signal. I'm not sure if those were needed.

I never checked to see if the signal was within or close to specs, but it was perfectly reliable once started. To start it, OUT15 had to be set for the desired length of the PWM cycle, and BLANK had to be pulsed high. Sometimes, it started by itself, but that was unreliable, especially because the initial contents of TLC5940 registers are undefined.

In up mode, all output modes are useful for TACCR0

The MSP430x2xx Family User's Guide, the section on Timer_A output modes claims that "Output modes 2, 3, 6, and 7 are not useful for output unit 0, because EQUx = EQU0". However, the output example for up mode states "The OUTx signal is changed when the timer counts up to the TACCRx value, and rolls from TACCR0 to zero, depending on the output mode." Based on this, it seems those output modes are useful: they could create a one cycle pulse. This does actually work; I used it to drive the BLANK input of a TLC5940.

Sunday, April 03, 2011

Video capture

I recently used VirtualDub 1.9.11 to capture video with audio.

I captured video via an ATI TV-Wonder VE card, which is based on the BrookTree/Conexant Bt878 chip. Windows 7 drivers are not available from Microsoft or the Manufacturer, but the version 5.3.8 btwincap open source driver works pretty well. In VirtualDub capture mode, overlay never works and resolutions can't be set via the capture pin, but preview works, and resolutions can be set via "Set custom format". Spending many minutes in (nonfunctional) overlay mode can cause a bluescreen or crash, but that's easy to avoid by switching to preview or "no display" mode. When previewing or capturing, the card is perfectly stable. Video quality is great.

The TV-Wonder VE doesn't capture audio, so I recorded audio via Realtek ALC889a line in with driver version 6.0.1.6235. The audio quality was good, but since audio and video were recorded via separate devices, they weren't in sync. My first capture was a major disappointment, perhaps due to Windows enhancements for line in. After I disabled those, things got better. I discovered that VirtualDub could perfectly sync if "Correct video timing for fewer frame drops/inserts" was not checked in "Capture timing options". If the option was checked, no frames were dropped and fewer frames were inserted, but VirtualDub did not synchronize the audio very well. There was both an offset and a drift. Fortunately, in most cases the drift was insignificant, and so I could simply correct for the offset. I captured most video this way to minimize frame drops and inserts.

I spent some time wondering whether to extend the luma black or white points. It seemed like there was information beyond the luma white point, but extending it didn't help, perhaps because that information was distorted. There was also a bit of information beyond the black point, but extending it increases noise. Extending either made properly exposed scenes seem washed out. I decided to never extend the white point and only extend the black point for a few scenes which would otherwise be excessively dark.

All the frame inserts I got happened before scene changes or severe noise. This makes me think that the card simply didn't record frames where a usable signal wasn't available. When an insert happened before a scene change, what followed was either one frame of the old scene followed by the first frame of the new scene, or one frame of the old scene and one frame of the old scene interlaced with the new scene. In such cases, I removed those frames.

I gave up on deinterlacing to 29.97fps because all algorithms are a compromise between blur and artifacts, and I didn't want to encode that permanently in the video. Deinterlacing to 59.94fps produced good results, but it also increased compressed video size substantially. Because of that, I decided to encode interlaced MBAFF H.264 video with x264. It plays acceptably in Windows Media Player 12 and excellently in MPC-HC with ffdshow Yadif deinterlacing.

Before encoding I considered various forms of denoising, but I never found one that really improves the end result. Denoising could decrease the slight noise, but it would also blur some subtle patterns like distant grass, asphalt, and slightly dirty walls. Also, the file size decrease from denoising is offset by the greater visibility of artifacts in smoothed areas. The most bothersome effect was that areas which show motion due to subtle patterns could stop showing motion. Finally, I noticed that x264 seems to perform some denoising itself, so denoising isn't needed after all.

Reversing a PAL to NTSC conversion

I wanted to capture video from an NTSC VHS tape that had been converted from a PAL tape. Of course, it would be better to capture from the original PAL tape, but I only had the NTSC tape and so I was forced to work from that.

The NTSC tape has some fields which have breaks in them. The area above the break is the next field, and the area before the break is the previous field. These fields cause a slight motion stutter, and motion is perfectly smooth without them. It's clear that they have been inserted to increase the field rate from PAL's 50 Hz to NTSC's 59.94 Hz.

Usually, the added fields occur every 6 fields. This means there are 5 original fields, 1 fake field, 5 original fields, and so on. However, the break in the field slowly moves down the field. When the break moves off the bottom of the field, then there are a few fields that are total duplicates. After that, the break moves into the top of the next field. Before that happens, it is necessary to skip ahead 7 fields once (outputting 6 original fields) to stay in sync with the inserted field. Another way to look at this is that there is a signal to go back one field which repeats at slightly more than 6 field times. It's not exactly 6 because NTSC uses 59.94 instead of 60 Hz, and so the ratio should be slightly less than 6/5.

Each inserted field flips the correspondence between PAL and NTSC odd and even fields, and in any case, PAL interlace can't really translate to NTSC interlace. (Imagine two overlapping combs with different teeth spacing.) Because of this, the converter had to deinterlace to 50 fps progressive, or more likely, just resize fields vertically while shifting them to compensate for vertical displacement due to interlace.

When outputting NTSC video, the converter bobbed the fields up and down to simulate interlace. However, this was done incorrectly. The image should bob up and down by one pixel at 480 lines or half a pixel at 240 lines, but instead, it bobbed up and down by one whole pixel at 240 lines. This finally made me give up on trying to recover interlaced video. Instead, I decided to capture at 320x480 and end up with 50 frames per second progressive video at 320x240.

The first processing step corrected for the bobbing. Using AviSynth, I added a 1 pixel border at the bottom of the top fields and cropped off 1 pixel from the top. This didn't lose any image data, because the top row in these fields was black.

Once the bobbing was corrected, I used AviSynth's RGBDifferenceFromPrevious function to collect data on inserted fields. By cropping the video so only a few lines at the top or bottom remain, I detected when the top part of the frame is the next frame or the bottom part is the previous frame. By using the function on the whole image, I detected when the switchover is in the vertical retrace interval, and frames are total duplicates. In all cases, it was necessary to crop off blackness, the very edges which are noisy, and the video head switching at the very bottom.

After a bit of experimentation, I chose to use data from frame tops, and to process it using a program which decides whether the duplicate frame occurs in 5, 6 or 7 frames from the previous one. It never occurs in 5 frames, but that capability allows the program to resynchronize if it chooses 7 when it should have chosen 6. After this, another simple program replaced the "7, 5" combinations with "6, 6" and counted the lengths of the spans of 6 that occurred before a 7.

The resulting data was good, but it had some glitches. I used a spreadsheet to work with it. There, I automatically removed some minor jitter and manually fixed a few larger glitches. When I tried to fit a line to the data, I found that it was actually a hockey stick curve with the bend at the start. This was probably because oscillators drifted during warmup and then stabilized. I considered trying to fit some kind of function to the graph, but minor fixes were sufficient.

Once I was satisfied with the data, I wrote another simple program which created a VirtualDub script. First, I had it keep the frames I wanted to remove, to ensure that I am indeed removing the frames which have tears in them. Then I created the final script which removed those frames.

After the video was finished, it was time to synchronize the audio. This can either be done by resampling the audio or changing the frame rate. I chose to resample the audio. For perfect sync, I could have used the data I generated to create a variable sample rate, but I instead just used a fixed sample rate based on a linear approximation. The errors were small enough to be unnoticeable.

Finally, it was time to encode the audio and video. I used a low (high quality) CRF in x264, because at 320x240 the video was quite sharp and I didn't want to degrade it.

Tuesday, March 15, 2011

Authorize.net makes it hard to get a refund after a credit card expires

I'm having difficulties obtaining a refund after the credit card expired. The merchant is willing to provide a refund, but Authorize.net doesn't allow it. After a conference call with the merchant and Authorize.net support, it turns out the merchant needs to add Expanded Credit Capabilities to their account. That service is free, but it involves filling out a form and that makes the merchant a bit reluctant.

This is happening even though the transaction was less than a month ago, the same credit card account is still active, and the new card has the same number. The merchant tried just updating the expiration date, but that didn't work.

Monday, March 14, 2011

Manually downloading Yahoo Messenger

If you "download Yahoo Messenger" from Yahoo, you actually download a small executable which can download Messenger and other things. Yahoo doesn't provide you with links to the actual executable which installs Messenger. You can get links from an .ini file you can download:

http://xp.yimg.com/gj/msgr/10/ini/ymsgr10_us.ini

For other countries, change the _us to another country code, such as _ca for Canada. The Messenger install file is located at the URL which is the INST_PATH value in the YMSGR section. Currently Yahoo Messenger is located at:

http://xh.yimg.com/gj/msgr/10/client/ymsgr1000_1270_us.exe

Friday, March 04, 2011

Copying Windows to a new hard drive with dd

Based on the existence of disk cloning software, one might assume that specialized software is needed to move Windows to a new hard drive. However, that's not true; the raw data can simply be copied. My normal personal setup has a small OS partition and a large data partition. I did the copying from Linux (SystemRescueCd) using dd. Here's the procedure I followed, with sda being the new drive and sdb being the old one (don't mix them up; that could cause data loss):
  1. Copy the start sectors, before the first partition: "dd if=/dev/sdb of=/dev/sda bs=512 count=63". I could have just copied the MBR with "count=1".
  2. Run fdisk and delete the data partition from the new drive. As a side effect, this causes Linux to recognize the newly written partition table.
  3. Copy the system partition to the new drive: "dd if=/dev/sdb1 of=/dev/sda1 bs=1M".
  4. Run fdisk and change the ID of the old drive. This is one of the advanced functions. There's a bug in fdisk: it doesn't save the partition table after this. To work around the bug, make another change. I set the type of the old system partition to NTFS hidden.
  5. Reboot into Windows. It should boot off the new drive and mount the partition there as the system partition.
  6. In Windows Disk Management, extend the new partition. Windows can always use more space, and new drives are usually bigger.
  7. Still in Disk Management, create the new data partition.
  8. Copy data to the new data partition.
  9. Remove the old hard drive. It is no longer needed.
If there's no need to resize the system partition and the new drive is the same size or bigger than the old drive, the entire drive can be copied, simply using "dd if=/dev/sdb of=/dev/sda bs=1M".

When working with modern large hard drives and not moving partitions relative to the start of the disk, the NT boot sector doesn't need to be altered. If moving a partition (for example, to align it for Advanced Format on the new drive) it would be necessary to change "Number of Hidden Sectors" at offset 0x1C in the boot sector.

If the new drive fails, and the old drive has to be used again, the procedure is simple: just change the disk ID to its former ID and unhide the system partition.

Medieval CUE Splitter 1.2 is broken!

Medieval CUE Splitter is a seemingly nice and user friendly Windows application which can split many audio formats using CUE files. It supports many audio formats and preserves tag information. In terms of features, it seems so great, but it's broken and it loses audio data!

I already knew that splitting of MP3 files is broken. I wrote about it in the past, with version 1.0. A quick check in version 1.2 shows that it still doesn't work correctly.

Recently, I learned that FLAC splitting is also broken. The total number of samples in the output files is less than the number of samples in the input file. This means data was lost. There is also another minor bug which may actually be helpful: the MD5 value is not written to the file. This can be used to identify FLAC files may have been split using Medieval CUE Splitter, because "flac -t" will complain about it.

So far, it seems that Monkey's Audio (APE) files split without any data loss. The number of samples is correct, and the concatenated audio data from the split files matches the audio data from the source file. This means that as a workaround, other lossless files can be converted to APE and split. However, this workaround certainly slows things down, and I can't really trust the program because of the other bugs.

Wednesday, January 12, 2011

Don't upgrade to PowerChute Personal Edition 3.0

If you're running Windows 7 or Vista and you have UAC enabled, don't upgrade to PowerChute Personal Edition 3.0. APC doesn't know how to properly program for Windows 7 and Vista. Their system tray application needs administrative rights, and so a UAC prompt will appear after every bootup or login. APC knows about this issue, and they suggest reducing your security settings as a workaround.

Administrative rights are requested by the executable's manifest. The manifest could be edited or even removed. However, that would probably cause problems. Just use an older version of PowerChute Personal Edition; it works fine.

Tuesday, January 04, 2011

Prolific PL-2303 driver 2.0.2.1 causes crashes

Version 2.0.2.1 of the Prolific PL-2303 USB serial port driver is buggy. It caused an IRQL_NOT_LESS_OR_EQUAL (0x1000000a) crash twice, both times when I was sending a file to a serial port via copy in cmd and I aborted the copy with Control-C. Windows fails to find an updated driver, but a much newer driver can be downloaded from Prolific. I'm now using version 3.3.11.152, which seems stable so far. This is in 32 bit Windows 7.

Sunday, January 02, 2011

Setting the MAC address on an Inspiron 6400

Dell Wireless 1500 802.11n WLAN Mini-card

The card is actually based on a Broadcom BCM4328 chip, and the drivers are from Broadcom. I'm currently using driver version 5.60.48.35. The advanced tab of properties in Device Manager allows a "Locally Administered MAC Address" to be assigned. It is an 8 byte (or 12 character) hexadecimal string using uppercase letters. It must be a locally administered unicast address, meaning the least significant bits of the first byte are 10 (and therefore the second character is 2, 6, A or E). An address which doesn't follow this format will be accepted without any error messages, but the driver will ignore it, and the card will continue to use its original address.

I wonder if the use of a locally administered address is being enforced to prevent copying of other addresses and circumvention of MAC address filtering on wireless networks. I also wonder if it would be easy to reverse-engineer the driver and remove this limitation. However, I currently have no need for this so I will not attempt it.

Broadcom 440x 10/100 Integrated Controller

I'm currently using driver version 4.60.0.1. There was no option for changing the MAC address in the advanced tab of properties in Device Manager. However, the address can be changed via the registry. The registry entry needs to be located within the adapter's numbered key in "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}", the same key where "DriverDesc" is "Broadcom 440x 10/100 Integrated Controller". Its name needs to be "NetworkAddress" and it must be of REG_SZ (string) type. The address must be entered as a hexadecimal number, with uppercase letters and no separators between bytes. After changing this entry, the adapter has to be disabled and re-enabled for it to take effect. (When changing the address via the properties dialog, Windows does this automatically.)

It's possible to add a "Locally Administered MAC Address" item to the advanced properties tab. All the items there are described by keys in the registry in the "Ndi\params" key within the adapter's key. Since the wireless card sets the MAC address in the same way, you can export the wireless card's "NetworkAddress" key, change the adapter number in the .reg file to the 440x, and then import the .reg file. Alternatively, use the following, also making sure that the number ("0007" here) matches the number of the 440x adapter's key:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}\0007\Ndi\params\NetworkAddress]
"ParamDesc"="Locally Administered MAC Address"
"type"="edit"
"LimitText"="12"
"UpperCase"="1"
"default"=""
"optional"="1"


The 440x only demands that you use a unicast address, meaning the least significant bit of the first byte is 0 (and therefore the second character is even). If the bit is not zero, the driver will ignore the address and continue using the original address. Unlike with the wireless card, the locally administered bit does not need to be set. This means the mac address of another device can be cloned or spoofed.