Category: Blog

In addition to the free forecast and observation data that powers Temperature.Express, the National Weather Service also makes available live and archive data from their NEXRAD weather radar network. This data dates back to 1991. I have created npm packages for parsing and plotting this data. [Jump to packages]

Level 2 Data

Level 2 data is what you’re used to seeing in your weather or radar app. It’s the “how hard is it raining” graphics typically in green, yellow and red with red being the heaviest or most severe precipitation. More advanced users will also recognize the red and green velocity plot that shows wind direction and intensity.

In 2019 I found nexrad-radar-data on Github. It worked, but needed some updating to to modernize the code and deal with the new compressed formats that were being stored to the archive. I then added plotting capabilities in a separate package. The results can be seen below.

Example reflectivity output from nexrad-level-2-plot
Reflectivity plot of hurricane Harvey on August 25, 2017
Example velocity output from nexrad-level-2-plot
Velocity plot of hurricane Harvey on August 25, 2017

A commercial tool exists for viewing this data, as does a set of libraries for Python. Neither of these options lent themselves to the environment where I was looking to deploy them. So I went ahead and began improving what was available to suit my needs.

nexrad-level-2-data is the direct descendant of nexrad-radar-data mentioned above and outputs a JSON object containing all of the data from one level 2 scan. An excerpt from this data is below. Additional information on how to make sense of this information including links to formal specifications from NWS are available in the Github repo.

{
  "elevation_angle": 0.4833984375,
  "channel_config": 2,
  "waveform_type": 1,
  "super_res_control": {
    "super_res": {
      "halfDegreeAzimuth": true,
      "quarterKm": true,
      "300km": false
    },
    "dual_pol": {
      "300km": true
    }
  },
  "surv_prf_number": 1,
  "surv_prf_pulse": 15,
  "azimuth_rate": 21.148681640625,
  "ref_threshold": 16,
  "vel_threshold": 16,
  "sw_threshold": 16,
  "diff_ref_threshold": 16,
  "diff_ph_threshold": 16,
  "cor_coeff_threshold": 16,
  "edge_angle_s1": 0,
  "prf_num_s1": 0,
  "prf_pulse_s1": 0,
  "supplemental_data": {
    "sails_cut": false,
    "sails_sequence": 0,
    "mrle_cut": false,
    "mrle_sequence": 0,
    "mpda_cut": false,
    "base_tilt_cut": false
  },
  "edge_angle_s2": 0,
  "prf_num_s2": 0,
  "prf_pulse_s2": 0,
  "ebc_angle": 0,
  "edge_angle_s3": 0,
  "prf_num_s3": 0,
  "prf_pulse_s3": 0,
  "reserved": 0
}

The changes were to the source repo were substantial. As I learned more about the newer radar formats it was necessary to go back and rework the core of the parser so that different message and compression types could be detected at the appropriate places. The data format is well documented but was designed in the 1980’s so there are a lot of less commonly used data structures compared to a modern system. The documentation makes significant use of “half-words” (16 bit values), and has a special floating point representation designed around the exact needs of the radar’s output. Some of the significant changes include:

  • Looking for compression headers and passing data to a gzip decoder. Several gzip blocks might be present within one file and the parser detects these additional blocks and decodes them accordingly. This is not a standard gzip file format.
  • In the mid 2010’s radar sites were upgraded with super resolution which more or less quadrupled the resolution of the data. This change puts all new data formats into the data files and the correct format needs to be determined before decoding can take place. The plotter also needs to identify the two types to scale the resulting images properly.
  • When using live data, it is delivered as “chunks” that are typically 1/6th of a complete scan per file. Various methods were developed to combine multiple independent files and recombine it with header data that only appears in the first file in the series.
  • The data coming from the radar is not perfect. After running the parser and plotter in a production environment for a year, several errors would repeat themselves. Careful analysis of the problematic files with a hex editor revealed truncated files, incorrect length pointers and missing slices of the scan. Best-effort methods were added to try and find the correct lengths, pointers and to flag missing slices.

Level 3 Data

Level 3 data is the result of compiling data across several minutes or hours, or text output of analysis algorithms that flag hail, rotation and other phenomenon. These different types of output are called products.

This data takes on a completely different format from level 2, and for several of the products does not even produce data that can be plotted. Below is the output of Digital Total Accumulation which acts as a digital rain gauge, estimating the amount of rain that has fallen during a thunderstorm event at any point within the radar’s area.

Example DTA output from nexrad-level-3-data
Digital Total Accumulation output from storms passing through the Chicago area on July 23, 2022

Even with the significantly different data format. A lot of techniques from the level 2 software could be leveraged to make decoding level 3 data a simpler task. Concepts like half-words and special floating point and fixed-point binary formats continued to exist.

Other data that is stored in the level 3 format is text-only and may be presented as the pre-formatted text shown below, or in a more structured form that lends itself to a JSON output.

                                STORM STRUCTURE
     RADAR ID 552   DATE/TIME 04:19:21/19:02:24   NUMBER OF STORM CELLS  24

   STORM      AZRAN      BASE     TOP    CELL BASED VIL    MAX REF    HEIGHT
     ID      DEG/NM       KFT     KFT       KG/M**2          DBZ        KFT

     D6      120/ 63    < 6.5    19.6          29             57       15.1
     P2      165/ 83    < 9.7    21.6          22             54       12.6
     T9      190/ 94    <11.7    19.9          21             53       11.7
     U3       78/100    <12.8    27.3          21             52       12.8
     H8      109/159    <27.5    33.5          20             49       27.5
     T6      125/138    <21.5    34.6          19             49       21.5
     C2      114/ 97    <12.2    21.0          19             52       12.2
     K8      201/133    <20.4    25.1          18             50       20.4
     L9      152/ 79    < 9.0    20.1          15             50        9.0
     X0      194/106    <14.1    23.3          14             50       14.1

Level 1 Data

What about level 1 data? Level 1 data is only published as part of select case studies and not on a continual basis like level 2 and level 3. It is the raw data that comes back from the radar antenna in analog form. Upon receipt, it is processed immediately at the radar site into the digital level 2 format that is distributed through the National Weather Service’s various channels.

Acquiring Data

The data is available freely through different cloud providers. A good starting point is Nexrad on AWS which at the time of this writing appears to have the most complete level 2 archives, live level 2 chunks and level 3 data. This AWS source also provides notifications for new data in some of the buckets.

Summary

The complete suite of four npm packages consists of:

Back in the 90s it took a little more effort and planning to get a weather forecast. You could look at the newspaper when it arrived. There was the local morning news. A few radio stations might have it at 10 and 40 after. But for me I would go to Local on the 8s on The Weather Channel. I’ve recreated this with WeatherStar 4000+ and you don’t have to wait until the 8s any more! Oh, and the forecasts are current, no reruns from the 90s.

WeatherStar 4000+ Screenshot

The technology was really impressive at the time. The Weather Channel was on cable and at that time talked about weather forecasts on a national scope. But on the 8’s a local forecast was inserted into what was otherwise a regular, nationally broadcast cable channel. Wow! And if you went on vacation and watched The Weather Channel it was a different local forecast for where you were.

There is a lot of nostalgia for these forecasts. First there’s TWC Classics which documents and discusses the history of The Weather Channel through screenshots and video clips. Then there’s the WS4000 Simulator which strives for accuracy in both look and information that is presented. It runs on desktop OSes.

This project is a fork of ws4kp and runs in a web browser. As a refresher course in JavaScript I’ve made it my own by updating the interface to use new techniques such as Classes, and JavaScript standard replacements for jQuery. The project does not strive for accuracy as much as the Simulator does. This is much more of a no-fuss, just get a forecast project. I also made one addition that wasn’t present in the Weather Channel’s original hardware: an hourly forecast for the next 24 hours replaces the travel forecast in the default lineup. You can follow it and view the source on Github.

New hourly forecast display

The project’s name comes from the line of hardware that The Weather Channel provided to your local cable office to show the forecast. WeatherSTAR includes the acronym Satellite Transponder Addressable Receiver. As the acronym implies all of the data came from the satellite feed, and was hidden in clever ways right along the video broadcast. The first three units I, II and III were used throughout the 80’s and could only display text information, no icons with sun and clouds and no current radar. In 1990 the 4000 was introduced and was used for most of the decade. It featured graphics depicting current and foretasted weather and initially a still local radar image. A few years later it was updated to show several animated frames of the local radar. Wikipedia has a lot more information about the WeatherSTAR lineup.

I’m excited to announce I am releasing a long-running side project of mine https://temperature.express. It’s built to get you a forecast quickly in a graphic format.

Example Temperature Express forecast

The basic code for it is open source and available on Github. A few excerpts from its FAQ help to explain the project:

What is this?

The National Weather Service provides a huge amount of forecast and climate data for free including an hour-by-hour forecast for the next 7 days. Unfortunately, getting it displayed in a concise format is not something they do well. The main graph attempts to display the most relevant parts of the forecast (high and low temperature, wind chill, heat index, clouds, precipitation and dewpoint) in an easy to digest format.

Why is this better than my weather app?

This web site is meant to be fast, fast, fast. If you add a bookmark to this web site to your home screen I’d be willing to guess that this web page loads faster than your weather app and gets you the information you’re looking for 2-3x faster every time.

I chose an intentional minimal design to leave as much space as possible for the data. I’m an engineer, I prefer accurate information over cute, bubbly buttons.

Why a graph?

It makes it quicker to answer the typical weather questions like: Will it be warmer or cooler tomorrow? – Just compare the heights of the red line. Is it going to rain tomorrow? – Look for the blue filled areas. Will the sun be out on Saturday? – Look for no grey filled area on Saturday.

The graph makes it much easier to convey how the weather will change during the day. Your favorite weather app, or search result card might simply say “Sunny, High of 80” today. But that tiny little bit of information might mask something important like the temperature dropping to 60 by 5 pm. That big temperature drop might mess up your evening plans if you were going to be outside.

If you’re interested in the exact values, just tap any point on the graph to get the exact value and the time for that value. Or, If you want to view most of the data as a table, you can get one by clicking the table icon on the menu. I think you’ll quickly see that the graph is a much easier to digest format.

Why do you work on this?

I’m a programmer. JavaScript, HTML, Node.js and other web technologies are not part of my day job, but seem to be where the innovation is happening. I use this and other side projects to keep myself up to date with them.

Syntax is structure of the programming language. It can be thought of as the verbs, nouns and parts of speech that make up English, or a foreign language. Syntax can typically fall under 3 different categories: High level, low level, and graphical.

High level is what is most common today. This includes languages such as C, Visual Basic, PHP and others. These languages use human-readable elements such as if, then, else, while and return to control the flow of a program. Many of these languages share enough common elements that a programmer familiar with one language can often look at code written in a different language an often have an understanding of what it accomplishes. However, each of these languages is unique and many of them are designed around a particular task or programming concept. A thorough understanding of this underlying concept can help a programmer to make a decision on which language to use to accomplish a particular task.

A graphical language is a high level language that is often represented in a flowchart form to facilitate human readability. Flowcharts are mainly made up of decision branches, and function blocks, not unlike ones that are used to describe a process in a user manual. These languages are typically programmed using drag and drop. Grab the block you need and drop it in the appropriate place on the screen. Each block often has a few parameters associated with it. For example a decision block will have a two input values and a comparison operator, not unlike an if statement in a text based language. From this block you can then branch to a yes or no function block. Function blocks, like decision blocks, also have inputs. A block that performs and addition function, for example, would have two inputs for the values being added, and one output for the result. These languages often become very complex to follow because it is often difficult to place comments within the flowchart structure to help another programmer understand exactly what is happening, unlike comments in a text based program which can be placed nearly anywhere. Additionally, as the program becomes more complex, the flow of the program can become very difficult to follow as it will quickly begin to cover several pages, and connecting blocks between these pages can often be a challenge depending on exact language used.

Another example of a graphical language is ladder logic. It is typically used in industrial control systems, and is based directly on electrical schematics that look a lot like rungs of a ladder. This design was intentional, as an electrician familiar with common electrical symbols could very easily transition to programming in a ladder language, as the symbols and functions of the symbols was essentially the same. Over time, the language has evolved to add instructions and symbols that do not have an electrical counterpart. Ladder based languages also have the distinction of running continuously, unlike the text languages also discussed in this section. Again, this design is intentional as it is modeled after electrical schematics and components, which behave like a continuously running program, as long as the machine is turned on.

Low level languages are ultimately what is run on a processor, and the other two types of languages discussed here are ultimately turned into a low level language before they are executed. Low level languages often use mnemonics that help to make the language readable to programmers. A few lines from a typical low level language might read as follows:

LD A, 10
INC A
JP A,Z

In the above example LD, INC and JP are all opcodes, and are actual instructions that the processor carries out. The values following the opcodes are parameters for each code. In this case A is a register, 10 is a constant and Z is a label to jump to. Each of these opcodes is turned into a binary value that is the actual sequence of bits that the processor must see to carry out the command entered. Even though this has been made easier to read through the use of mnemonics, at first glance it is not entirely clear that this code loads the value 10 into register A, increments the value in A by 1 and then jumps to label Z if A is not zero. Programming in an assembly language requires a good understanding of the actual architecture of the processor, how memory is laid out, what functions it can perform and how to combine the rather primitive functions provided into more common functions such as if-then-else statements.

There are two basic ways that a program can be executed: continuously and event driven. The execution style is dictated by the programming language, and is almost always event driven.

In event driven code, nothing runs until there is a triggering event. This is often the user clicking a button, pressing a key, following a link on a web site, or it could be a machine generated event, such as data arrives on a port or a set amount of time has elapsed. In any case, once the triggering event happens, a portion of code is executed. In a calculator, for example, when the user presses the equals button, the corresponding portion of the program to add, subtract, multiply or divide is carried out.

Continuously executed programs run every line of code, from top to bottom, and then immediately begin executing at the top again. Every line of code has a distinct output that is updated on a more-or-less continuous basis, as long as the program and processor run sufficiently fast. The most basic type of this program is often referred to as “bit banging” and is often used in the programming of industrial equipment. It is called bit banging because these types of programs look at inputs from multiple single bit sources, often push buttons, position sensors or selector switches and based on the states of these inputs, turn on or off a single bit output. The code can often be broken down into several lines of if-then-else statements such as “if A or B are ON then turn ON C else turn OFF C.”

Interestingly, these two different styles of program execution, even though they are defined by the programming language, can be written in such a way that they behave as they other type of program. Consider the following lines of pseudo-code that are written in an event driven language:

Event (User Click)
While (1)
If A or B Then C Else D
If A and B Then E Else F
End While
End Event

The above program will execute continuously, even though the programming language is event driven. Strictly speaking, this code would “hang” the processor as it would be stuck in an infinite loop that only deals with inputs A and B. In reality, there is a supervisory program (an operating system in a computer, for example) that oversees everything that is going on and will pause the execution of this program and give processing time to other programs to keep everything running as expected.

Like the above example, a continuous execution program can be written to appear to execute as though it were event driven. Here is another pseudo-code example written in a continuous language.

If A then B
If B then C
If C then D

Even though all of these lines of code are executed over and over, the only way to get to D is to first trigger A, which triggers B and so on, in an event driven fashion.

One other interesting note, excluding the newer multi-core processor systems, any given processor can only execute one command at a time, and can only execute commands sequentially. The speed of a processor allows it to appear to handle multiple tasks at one.