The demo shows level 2 plots for reflectivity and velocity and level 3 data for the hybrid hydrometer classification. It also includes a timestamp and road/county overlay on the radar image for reference, but those items are not part of the actual plotting libraries. The plots shown on the demo web site are updated in real-time, but a page refresh is required to load the latest image.
Earlier this year I released the WeatherStar 4000+, a retro-looking weather forecast and current weather conditions app based on the look and feel of the 90’s Weather Channel.
We’re all used to weather information that’s very easy to access and is at our fingertips. My phone right now, in it’s “off” state, has a little blurb on screen with a few weather details. In the 30+ years since the WeatherStar debuted on cable both the accessibility and the amount of data available for a forecast is much more expansive.
For this project and for me the challenge and question has been can I integrate new forecast information and not disrupt the nostalgia of the system? After a lot of though earlier this year, I took a cue from the “Travel Forecast” scrolling screen and turned it into an hourly forecast in the same style. It fit within the aesthetic and worked very well to show off new hour-by-hour forecast data that is now available.
But I personally never liked the long scrolling screen. It just took too long to get all the way through it. So inspired by first several weather apps that provide a graph for the hourly forecast, and then another of my projects Tempreature.Express, I developed some new graphics for the WeatherStar4000+ for this exact purpose.
It’s new, and it’s not part of the original system. But I feel that it very well captures the low-resolution 90’s design of the original. A lot of considerations around how much data to display and how to arrange it were made. I borrowed as much as possible from already existing graphics such as:
The x-axis times borrow sizing color and location concept from the Current Conditions and Travel Forecast screens.
The legend in the top right is borrowed from the Radar screen.
The line graphics are inspired by the Marine Forecast display that isn’t present in this WeatherStar4000+ but is in other adaptations as well as the original.
The blue background surrounded by orange is used on many of the text based displays although slightly modified at the bottom for the x-axis labels.
The rest of the header and footer is common to almost all of the other displays in the original WeatherStar 4000.
Also part of my considerations were should I intentionally pixelate the graph? I decided against this. In a previous update to the WeatherStar 4000+ I changed from drawing on a canvas, which resulted in a pixelated look when used full-screen, to HTML and appropriate fonts. This means that everything appears clean and crisp when running full screen. The weather icons (such as “Sunny”) are still the same size and do appear slightly pixelated full-screen, but that’s part of the charm and nostalgia. I did go through the exercise of pixelating this new display by drawing it at half-resolution and then scaling up but it just looked bad against the background and text that all scales fairly cleanly. I backed out that change.
Please join the discussion at GitHub if you have questions or comments on the new screen.
The scanner will attempt to connect to ports at your IP address from netbymatt.com’s netowork and present results in a friendly grid format. All green “stealth” results are great!
The scanner has several uses besides overall security
Test to make sure a port is open, such as a VPN or web server
Test your network’s IPv6 connectivity. The scanner link above will attempt to connect to on both IPv6 and IPv4 and will present to you both addresses if both are found. You can then select which address to scan.
Finding your IP address remotely
Testing a headless server – The returned HTML is in a very straight-forward format and can be easily scanned by a human after running curl or similar from your server.
There’s a lot of technology behind the scenes to make this work. The link above has full details but some of the highlights include:
Scan originates from a different address than netbymatt.com
Micro-service architecture allows for easy scaling during traffic spikes
Weather websites at the time typically had a static image showing the radar, or maybe an animated GIF. This was great. You could see the current radar without having to wait for the 5pm news or for 8 minutes after on The Weather Channel. But for someone who was interested in weather this was still lacking.
My first foray into this was through a technique called screen scraping. On my web server I would run a PHP script every few minutes that would check my two favorite radar images at the time: Accuweather and Wunderground. That script would determine if a new image was present and store it to my server. It would keep a total of 5 or 10 images expiring the oldest when a new one arrived.
The web page associated with it would let you view either set of images as an animation by changing out the src attribute on an image tag. This method had it’s drawbacks that have all been addressed in subsequent updates.
Some drawbacks were that images weren’t pre-loaded. On a dial up modem, not uncommon at the time, it might take 3 or 4 times through the loop before all of the images loaded.
Improving the interface
The first two changes implemented allowed for changing the frame rate and and manual forward/back control of the images. Specifically, forward and backward control was accomplished by using the mouse’s scroll wheel. Later when touch screens became prevalent, dragging across the image was added as another way to control this.
This easy manual control via the scroll wheel is something that I have always felt was lacking, and is still, in any radar viewer that I’ve used. For me, this control is critical as an amateur weather enthusiast to be able to follow the storm’s track and watch it’s development.
Next I began pre-loading images. Initially this was done inside of a hidden div, although the method has changed over time. This meant that, especially on a slower connection, that the second and later frames might already be loaded when the animation reached them.
It also allowed for switching between the Wunderground and Accuweather image sets without reloading the page. And as the image sets grew to include TDWR, satellite, velocity and other regional and national images this became very helpful.
It was around this time as well that I began time stamping images. Or more specifically I would log metadata about the images to a database that included the time I downloaded the image. This was a reasonable estimate of the radar’s time. But it began to make a problem clear. Several of my image sets would occasionally get out of order. I never did sort out exactly why, and it didn’t happen on the source web sites. My suspicion was issues with caching at CDNs or connecting to a different CDN server on subsequent calls that was slightly out of date. This order-of-images problem lived on for almost 10 years.
The backend at this time continued to run on PHP. But around 2015 I started experimenting with Jquery and found that it was a much better way to handle all of the interactivity that I had added to the web page. I recall having so much code specific to Firefox, Chrome and at one point even IE and Jquery did away with this. It also integrated some really simple and helpful transition animations that very much helped show the process flow through the web site.
My Own Images
The journey here took a lot of effort. I’ll list some of the problems that had to be overcome:
The bzip format used in current radar products was not standard. There were headers interspersed with chunks of bzip data. The headers were not well defined but I eventually was able to sort them out and wrangle a nodejs bzip library to work on this data.
The data could be presented in an older format or the new dual-pol/hi-res format. These needed separate handling at the appropriate places to parse the data successfully. Separately, the plotting tool needed to understand the two types to be able to correctly show the data.
Plotting was slow, initially. A lot of work was done to speed up the plotting process. On a day with some storms, plotting went from 10s originally down to about 2s.
“Live” data, called chunks, took extra effort to process. A single image may come in as 3 or 6 chunks. I had to study a lot of raw radar data to determine whether to trigger on the 3rd or 6th chunk, and if there was another lowest-elevation scan at some other chunk.
I enjoyed the challenge of making all of this work and it solved the problem of out-of-order images that the screen-scraping method occasionally produced.
After I had updated the libraries to deal with the new data formats I had to find a way to run these. Previously I had used PHP on my web server that also hosted wordpress. But the new libraries ran on nodejs. This is when I discovered Lambda and it’s ability to run a one-off function with code that I supplied.
So I set about implementing the library in a way that a lambda function could handle, and it was very capable of plotting this data and storing it to S3. Then other metadata and tasks were adapted to the Lambda environment. The final stack looks something like this:
An S3 triggered Lambda that reads the first chunk in the set of chunks as they arrive and determines what chunks should be decoded. When those other chunks arrive it then triggers the next process.
A plot worker receives the “interesting” chunk numbers from the previous function, downloads all of the data for each chunk, plots it and stores it to S3. The plotting consists of reflectivity, velocity at various resolutions and crops.
The plot worker sends a command to a third lambda function that writes metadata to the database for retrieval by the fronted.
A separate backend tool that monitors the api.weather.gov for current severe thunderstorm and tornado warnings and stores these to a database in a format that is practical for displaying on the radar map.
A frontend “list” api that can return a list of current images for a selected radar site, and that can list all current warnings for the current site.
The frontend queries the API for the list of images and warnings, loads the images from S3 and draws the warnings on the image.
Radar images by themselves are not hugely helpful. You need some context to view them within. Typically road, state and county maps are shown over the radar plot. I used Open Street Map to provide major highways, county and state borders as a base-image for my radar plots. I downloaded the relevant data and used my own nodejs tool to draw these base maps from the data.
I hope you have enjoyed this journey and would like to see the resulting web site. However I am keeping that site private. There are two reasons for this. One historical and one is cost. Historically, during the screen-scraping days I am sure there were copyright issues with me copying and re-displaying radar images from other web sites. So I kept the site private.
Today, the images are not copyrighted. In fact, any data produced by an agency of the United States Government can not be copyrighted. But there is now a non-zero cost with producing the images. For me to plot the handful of radars around my location, it’s reasonable. But if I were to open the site to the public I would need to plot every radar site. And I’ve calculated this at $150-200 per month which is well outside of what I could pay for out of pocket.
It’s unfortunate, but I’m still very happy with my own results. And I do give my own web site a lot of traffic especially on days when there’s significant weather in my area.
In addition to the free forecast and observation data that powers Temperature.Express, the National Weather Service also makes available live and archive data from their NEXRAD weather radar network. This data dates back to 1991. I have created npm packages for parsing and plotting this data. [Jump to packages]
Level 2 Data
Level 2 data is what you’re used to seeing in your weather or radar app. It’s the “how hard is it raining” graphics typically in green, yellow and red with red being the heaviest or most severe precipitation. More advanced users will also recognize the red and green velocity plot that shows wind direction and intensity.
In 2019 I found nexrad-radar-data on Github. It worked, but needed some updating to to modernize the code and deal with the new compressed formats that were being stored to the archive. I then added plotting capabilities in a separate package. The results can be seen below.
A commercial tool exists for viewing this data, as does a set of libraries for Python. Neither of these options lent themselves to the environment where I was looking to deploy them. So I went ahead and began improving what was available to suit my needs.
nexrad-level-2-data is the direct descendant of nexrad-radar-data mentioned above and outputs a JSON object containing all of the data from one level 2 scan. An excerpt from this data is below. Additional information on how to make sense of this information including links to formal specifications from NWS are available in the Github repo.
The changes were to the source repo were substantial. As I learned more about the newer radar formats it was necessary to go back and rework the core of the parser so that different message and compression types could be detected at the appropriate places. The data format is well documented but was designed in the 1980’s so there are a lot of less commonly used data structures compared to a modern system. The documentation makes significant use of “half-words” (16 bit values), and has a special floating point representation designed around the exact needs of the radar’s output. Some of the significant changes include:
Looking for compression headers and passing data to a gzip decoder. Several gzip blocks might be present within one file and the parser detects these additional blocks and decodes them accordingly. This is not a standard gzip file format.
In the mid 2010’s radar sites were upgraded with super resolution which more or less quadrupled the resolution of the data. This change puts all new data formats into the data files and the correct format needs to be determined before decoding can take place. The plotter also needs to identify the two types to scale the resulting images properly.
When using live data, it is delivered as “chunks” that are typically 1/6th of a complete scan per file. Various methods were developed to combine multiple independent files and recombine it with header data that only appears in the first file in the series.
The data coming from the radar is not perfect. After running the parser and plotter in a production environment for a year, several errors would repeat themselves. Careful analysis of the problematic files with a hex editor revealed truncated files, incorrect length pointers and missing slices of the scan. Best-effort methods were added to try and find the correct lengths, pointers and to flag missing slices.
Level 3 Data
Level 3 data is the result of compiling data across several minutes or hours, or text output of analysis algorithms that flag hail, rotation and other phenomenon. These different types of output are called products.
This data takes on a completely different format from level 2, and for several of the products does not even produce data that can be plotted. Below is the output of Digital Total Accumulation which acts as a digital rain gauge, estimating the amount of rain that has fallen during a thunderstorm event at any point within the radar’s area.
Even with the significantly different data format. A lot of techniques from the level 2 software could be leveraged to make decoding level 3 data a simpler task. Concepts like half-words and special floating point and fixed-point binary formats continued to exist.
Other data that is stored in the level 3 format is text-only and may be presented as the pre-formatted text shown below, or in a more structured form that lends itself to a JSON output.
What about level 1 data? Level 1 data is only published as part of select case studies and not on a continual basis like level 2 and level 3. It is the raw data that comes back from the radar antenna in analog form. Upon receipt, it is processed immediately at the radar site into the digital level 2 format that is distributed through the National Weather Service’s various channels.
The data is available freely through different cloud providers. A good starting point is Nexrad on AWS which at the time of this writing appears to have the most complete level 2 archives, live level 2 chunks and level 3 data. This AWS source also provides notifications for new data in some of the buckets.
The complete suite of four npm packages consists of: