Planet Fellowship (en)

Friday, 26 May 2017

Last week of GSoC Community Bonding

vanitasvitae's blog » englisch | 19:12, Friday, 26 May 2017

This is my report for the last week of community bonding. On next tuesday the coding phase officially begins \o/.

I spent my week like I did the one before; writing tests, increasing the codecoverage of my Smack OMEMO module. Today the coverage finally reached the same level as the main codebase, meaning my PR wouldn’t decrease the percentage anymore. Apart from that I’ve read some tips on java.nio usage for file transfer and made myself more familiar with its non-blocking IO concepts.

Throughout the week I took the one or other chance to distract myself from work by pariticipating in OMEMO related discussions on the standards mailing list. I’m quite happy, that there’s a vital discussion on the topic which seems to have a solution in prospect which is acceptable for everyone. Specifying the OMEMO XEP in a way that enables implementations using different crypto libraries is definitely a huge step forward which might bring a broader adoption without leaving those who pioneered and developed the standard standing in the rain (all my subjective opinion). I was really surprised to see developers of the Matrix project participating in the discussion. That reminded me of what the spirit of floss software really is :)

I plan to spent the last days before the coding phase sketching out my projects structure and relaxing a little before the hard work begins. One of my goals is to plan ahead and I really hope to fulfill this goal.

Happy Hacking :)

Vanitasvitae

Monday, 22 May 2017

VGA Signal Generation with the PIC32

Paul Boddie's Free Software-related blog » English | 15:08, Monday, 22 May 2017

It all started after I had designed – and received from fabrication – a circuit board for prototyping cartridges for the Acorn Electron microcomputer. Although some prototyping had already taken place with an existing cartridge, with pins intended for ROM components being routed to drive other things, this board effectively “breaks out” all connections available to a cartridge that has been inserted into the computer’s Plus 1 expansion unit.

Acorn Electron cartridge breakout board

The Acorn Electron cartridge breakout board being used to drive an external circuit

One thing led to another, and soon my brother, David, was interfacing a microcontroller to the Electron in order to act as a peripheral being driven directly by the system’s bus signals. His approach involved having a program that would run and continuously scan the signals for read and write conditions and then interpret the address signals, sending and receiving data on the bus when appropriate.

Having acquired some PIC32 devices out of curiosity, with the idea of potentially interfacing them with the Electron, I finally took the trouble of looking at the datasheet to see whether some of the hard work done by David’s program might be handled by the peripheral hardware in the PIC32. The presence of something called “Parallel Master Port” was particularly interesting.

Operating this function in the somewhat insensitively-named “slave” mode, the device would be able to act like a memory device, with the signalling required by read and write operations mostly being dealt with by the hardware. Software running on the PIC32 would be able to read and write data through this port and be able to get notifications about new data while getting on with doing other things.

So began my journey into PIC32 experimentation, but this article isn’t about any of that, mostly because I put that particular investigation to one side after a degree of experience gave me perhaps a bit too much confidence, and I ended up being distracted by something far more glamorous: generating a video signal using the PIC32!

The Precedents’ Hall of Fame

There are plenty of people who have written up their experiments generating VGA and other video signals with microcontrollers. Here are some interesting examples:

And there are presumably many more pages on the Web with details of people sending pixel data over a cable to a display of some sort, often trying to squeeze every last cycle out of their microcontroller’s instruction processing unit. But, given an awareness of how microcontrollers should be able to take the burden off the programs running on them, employing peripheral hardware to do the grunt work of switching pins on and off at certain frequencies, maybe it would be useful to find examples of projects where such advantages of microcontrollers had been brought to bear on the problem.

In fact, I was already aware of the Maximite “single chip computer” partly through having seen the cloned version of the original being sold by Olimex – something rather resented by the developer of the Maximite for reasons largely rooted in an unfortunate misunderstanding of Free Software licensing on his part – and I was aware that this computer could generate a VGA signal. Indeed, the method used to achieve this had apparently been written up in a textbook for the PIC32 platform, albeit generating a composite video signal using one of the on-chip SPI peripherals. The Colour Maximite uses three SPI channels to generate one red, one green, and one blue channel of colour information, thus supporting eight-colour graphical output.

But I had been made aware of the Parallel Master Port (PMP) and its “master” mode, used to drive LCD panels with eight bits of colour information per pixel (or, using devices with many more pins than those I had acquired, with sixteen bits of colour information per pixel). Would it surely not be possible to generate 256-colour graphical output at the very least?

Information from people trying to use PMP for this purpose was thin on the ground. Indeed, reading again one article that mentioned an abandoned attempt to get PMP working in this way, using the peripheral to emit pixel data for display on a screen instead of a panel, I now see that it actually mentions an essential component of the solution that I finally arrived at. But the author had unfortunately moved away from that successful component in an attempt to get the data to the display at a rate regarded as satisfactory.

Direct Savings

It is one thing to have the means to output data to be sent over a cable to a display. It is another to actually send the data efficiently from the microcontroller. Having contemplated such issues in the past, it was not a surprise that the Maximite and other video-generating solutions use direct memory access (DMA) to get the hardware, as opposed to programs, to read through memory and to write its contents to a destination, which in most cases seemed to be the memory address holding output data to be emitted via a data pin using the SPI mechanism.

I had also envisaged using DMA and was still fixated on using PMP to emit the different data bits to the output circuit producing the analogue signals for the display. Indeed, Microchip promotes the PMP and DMA combination as a way of doing “low-cost controllerless graphics solutions” involving LCD panels, so I felt that there surely couldn’t be much difference between that and getting an image on my monitor via a few resistors on the breadboard.

And so, a tour of different PIC32 features began, trying to understand the DMA documentation, the PMP documentation, all the while trying to get a grasp of what the VGA signal actually looks like, the timing constraints of the various synchronisation pulses, and battle various aspects of the MIPS architecture and the PIC32 implementation of it, constantly refining my own perceptions and understanding and learning perhaps too often that there may have been things I didn’t know quite enough about before trying them out!

Using VGA to Build a Picture

Before we really start to look at a VGA signal, let us first look at how a picture is generated by the signal on a screen:

VGA Picture Structure

The structure of a display image or picture produced by a VGA signal

The most important detail at this point is the central area of the diagram, filled with horizontal lines representing the colour information that builds up a picture on the display, with the actual limits of the screen being represented here by the bold rectangle outline. But it is also important to recognise that even though there are a number of visible “display lines” within which the colour information appears, the entire “frame” sent to the display actually contains yet more lines, even though they will not be used to produce an image.

Above and below – really before and after – the visible display lines are the vertical back and front porches whose lines are blank because they do not appear on the screen or are used to provide a border at the top and bottom of the screen. Such extra lines contribute to the total frame period and to the total number of lines dividing up the total frame period.

Figuring out how many lines a display will have seems to involve messing around with something called the “generalised timing formula”, and if you have an X server like Xorg installed on your system, you may even have a tool called “gtf” that will attempt to calculate numbers of lines and pixels based on desired screen resolutions and frame rates. Alternatively, you can look up some common sets of figures on sites providing such information.

What a VGA Signal Looks Like

Some sources show diagrams attempting to describe the VGA signal, but many of these diagrams are open to interpretation (in some cases, very much so). They perhaps show the signal for horizontal (display) lines, then other signals for the entire image, but they either do not attempt to combine them, or they instead combine these details ambiguously.

For instance, should the horizontal sync (synchronisation) pulse be produced when the vertical sync pulse is active or during the “blanking” period when no pixel information is being transmitted? This could be deduced from some diagrams but only if you share their authors’ unstated assumptions and do not consider other assertions about the signal structure. Other diagrams do explicitly show the horizontal sync active during vertical sync pulses, but this contradicts statements elsewhere such as “during the vertical sync period the horizontal sync must also be held low”, for instance.

After a lot of experimentation, I found that the following signal structure was compatible with the monitor I use with my computer:

VGA Signal Structure

The basic structure of a VGA signal, or at least a signal that my monitor can recognise

There are three principal components to the signal:

  • Colour information for the pixel or line data forms the image on the display and it is transferred within display lines during what I call the visible display period in every frame
  • The horizontal sync pulse tells the display when each horizontal display line ends, or at least the frequency of the lines being sent
  • The vertical sync pulse tells the display when each frame (or picture) ends, or at least the refresh rate of the picture

The voltage levels appear to be as follows:

  • Colour information should be at 0.7V (although some people seem to think that 1V is acceptable as “the specified peak voltage for a VGA signal”)
  • Sync pulses are supposed to be at “TTL” levels, which apparently can be from 0V to 0.5V for the low state and from 2.7V to 5V for the high state

Meanwhile, the polarity of the sync pulses is also worth noting. In the above diagram, they have negative polarity, meaning that an active pulse is at the low logic level. Some people claim that “modern VGA monitors don’t care about sync polarity”, but since it isn’t clear to me what determines the polarity, and since most descriptions and demonstrations of VGA signal generation seem to use negative polarity, I chose to go with the flow. As far as I can tell, the gtf tool always outputs the same polarity details, whereas certain resources provide signal characteristics with differing polarities.

It is possible, and arguably advisable, to start out trying to generate sync pulses and just grounding the colour outputs until your monitor (or other VGA-capable display) can be persuaded that it is receiving a picture at a certain refresh rate and resolution. Such confirmation can be obtained on a modern display by seeing a blank picture without any “no signal” or “input not supported” messages and by being able to activate the on-screen menu built into the device, in which an option is likely to exist to show the picture details.

How the sync and colour signals are actually produced will be explained later on. This section was merely intended to provide some background and gather some fairly useful details into one place.

Counting Lines and Generating Vertical Sync Pulses

The horizontal and vertical sync pulses are each driven at their own frequency. However, given that there are a fixed number of lines in every frame, it becomes apparent that the frequency of vertical sync pulse occurrences is related to the frequency of horizontal sync pulses, the latter occurring once per line, of course.

With, say, 622 lines forming a frame, the vertical sync will occur once for every 622 horizontal sync pulses, or at a rate that is 1/622 of the horizontal sync frequency or “line rate”. So, if we can find a way of generating the line rate, we can not only generate horizontal sync pulses, but we can also count cycles at this frequency, and every 622 cycles we can produce a vertical sync pulse.

But how do we calculate the line rate in the first place? First, we decide what our refresh rate should be. The “classic” rate for VGA output is 60Hz. Then, we decide how many lines there are in the display including those extra non-visible lines. We multiply the refresh rate by the number of lines to get the line rate:

60Hz * 622 = 37320Hz = 37.320kHz

On a microcontroller, the obvious way to obtain periodic events is to use a timer. Given a particular frequency at which the timer is updated, a quick calculation can be performed to discover how many times a timer needs to be incremented before we need to generate an event. So, let us say that we have a clock frequency of 24MHz, and a line rate of 37.320kHz, we calculate the number of timer increments required to produce the latter from the former:

24MHz / 37.320kHz = 24000000Hz / 37320Hz = 643

So, if we set up a timer that counts up to 642 and then upon incrementing again to 643 actually starts again at zero, with the timer sending a signal when this “wraparound” occurs, we can have a mechanism providing a suitable frequency and then make things happen at that frequency. And this includes counting cycles at this particular frequency, meaning that we can increment our own counter by 1 to keep track of display lines. Every 622 display lines, we can initiate a vertical sync pulse.

One aspect of vertical sync pulses that has not yet been mentioned is their duration. Various sources suggest that they should last for only two display lines, although the “gtf” tool specifies three lines instead. Our line-counting logic therefore needs to know that it should enable the vertical sync pulse by bringing it low at a particular starting line and then disable it by bringing it high again after two whole lines.

Generating Horizontal Sync Pulses

Horizontal sync pulses take place within each display line, have a specific duration, and they must start at the same time relative to the start of each line. Some video output demonstrations seem to use lots of precisely-timed instructions to achieve such things, but we want to use the peripherals of the microcontroller as much as possible to avoid wasting CPU time. Having considered various tricks involving specially formulated data that might be transferred from memory to act as a pulse, I was looking for examples of DMA usage when I found a mention of something called the Output Compare unit on the PIC32.

What the Output Compare (OC) units do is to take a timer as input and produce an output signal dependent on the current value of the timer relative to certain parameters. In clearer terms, you can indicate a timer value at which the OC unit will cause the output to go high, and you can indicate another timer value at which the OC unit will cause the output to go low. It doesn’t take much imagination to realise that this sounds almost perfect for generating the horizontal sync pulse:

  1. We take the timer previously set up which counts up to 643 and thus divides the display line period into units of 1/643.
  2. We identify where the pulse should be brought low and present that as the parameter for taking the output low.
  3. We identify where the pulse should be brought high and present that as the parameter for taking the output high.

Upon combining the timer and the OC unit, then configuring the output pin appropriately, we end up with a low pulse occurring at the line rate, but at a suitable offset from the start of each line.

VGA Display Line Structure

The structure of each visible display line in the VGA signal

In fact, the OC unit also proves useful in actually generating the vertical sync pulses, too. Although we have a timer that can tell us when it has wrapped around, we really need a mechanism to act upon this signal promptly, at least if we are to generate a clean signal. Unfortunately, handling an interrupt will introduce a delay between the timer wrapping around and the CPU being able to do something about it, and it is not inconceivable that this delay may vary depending on what the CPU has been doing.

So, what seems to be a reasonable solution to this problem is to count the lines and upon seeing that the vertical sync pulse should be initiated at the start of the next line, we can enable another OC unit configured to act as soon as the timer value is zero. Thus, upon wraparound, the OC unit will spring into action and bring the vertical sync output low immediately. Similarly, upon realising that the next line will see the sync pulse brought high again, we can reconfigure the OC unit to do so as soon as the timer value again wraps around to zero.

Inserting the Colour Information

At this point, we can test the basic structure of the signal and see if our monitor likes it. But none of this is very interesting without being able to generate a picture, and so we need a way of getting pixel information from the microcontroller’s memory to its outputs. We previously concluded that Direct Memory Access (DMA) was the way to go in reading the pixel data from what is usually known as a framebuffer, sending it to another place for output.

As previously noted, I thought that the Parallel Master Port (PMP) might be the right peripheral to use. It provides an output register, confusingly called the PMDIN (parallel master data in) register, that lives at a particular address and whose value is exposed on output pins. On the PIC32MX270, only the least significant eight bits of this register are employed in emitting data to the outside world, and so a DMA destination having a one-byte size, located at the address of PMDIN, is chosen.

The source data is the framebuffer, of course. For various retrocomputing reasons hinted at above, I had decided to generate a picture 160 pixels in width, 256 lines in height, and with each byte providing eight bits of colour depth (specifying how many distinct colours are encoded for each pixel). This requires 40 kilobytes and can therefore reside in the 64 kilobytes of RAM provided by the PIC32MX270. It was at this point that I learned a few things about the DMA mechanisms of the PIC32 that didn’t seem completely clear from the documentation.

Now, the documentation talks of “transactions”, “cells” and “blocks”, but I don’t think it describes them as clearly as it could do. Each “transaction” is just a transfer of a four-byte word. Each “cell transfer” is a collection of transactions that the DMA mechanism performs in a kind of batch, proceeding with these as quickly as it can until it either finishes the batch or is told to stop the transfer. Each “block transfer” is a collection of cell transfers. But what really matters is that if you want to transfer a certain amount of data and not have to keep telling the DMA mechanism to keep going, you need to choose a cell size that defines this amount. (When describing this, it is hard not to use the term “block” rather than “cell”, and I do wonder why they assigned these terms in this way because it seems counter-intuitive.)

You can perhaps use the following template to phrase your intentions:

I want to transfer <cell size> bytes at a time from a total of <block size> bytes, reading data starting from <source address>, having <source size>, and writing data starting at <destination address>, having <destination size>.

The total number of bytes to be transferred – the block size – is calculated from the source and destination sizes, with the larger chosen to be the block size. If we choose a destination size less than the source size, the transfers will not go beyond the area of memory defined by the specified destination address and the destination size. What actually happens to the “destination pointer” is not immediately obvious from the documentation, but for our purposes, where we will use a destination size of one byte, the DMA mechanism will just keep writing source bytes to the same destination address over and over again. (One might imagine the pointer starting again at the initial start address, or perhaps stopping at the end address instead.)

So, for our purposes, we define a “cell” as 160 bytes, being the amount of data in a single display line, and we only transfer one cell in a block. Thus, the DMA source is 160 bytes long, and even though the destination size is only a single byte, the DMA mechanism will transfer each of the source bytes into the destination. There is a rather unhelpful diagram in the documentation that perhaps tries to communicate too much at once, leading one to believe that the cell size is a factor in how the destination gets populated by source data, but the purpose of the cell size seems only to be to define how much data is transferred at once when a transfer is requested.

DMA Transfer Mechanism

The transfer of framebuffer data to PORTB using DMA cell transfers (noting that this hints at the eventual approach which uses PORTB and not PMDIN)

In the matter of requesting a transfer, we have already described the mechanism that will allow us to make this happen: when the timer signals the start of a new line, we can use the wraparound event to initiate a DMA transfer. It would appear that the transfer will happen as fast as both the source and the destination will allow, at least as far as I can tell, and so it is probably unlikely that the data will be sent to the destination too quickly. Once the transfer of a line’s pixel data is complete, we can do some things to set up the transfer for the next line, like changing the source data address to point to the next 160 bytes representing the next display line.

(We could actually set the block size to the length of the entire framebuffer – by setting the source size – and have the DMA mechanism automatically transfer each line in turn, updating its own address for the current line. However, I intend to support hardware scrolling, where the address of the first line of the screen can be adjusted so that the display starts part way through the framebuffer, reaches the end of the framebuffer part way down the screen, and then starts again at the beginning of the framebuffer in order to finish displaying the data at the bottom of the screen. The DMA mechanism doesn’t seem to support the necessary address wraparound required to manage this all by itself.)

Output Complications

Having assumed that the PMP peripheral would be an appropriate choice, I soon discovered some problems with the generated output. Although the data that I had stored in the RAM seemed to be emitted as pixels in appropriate colours, there were gaps between the pixels on the screen. Yet the documentation seemed to vaguely indicate that the PMDIN register was more or less continuously updated. That meant that the actual output signals were being driven low between each pixel, causing black-level gaps and ruining the result.

I wondered if anything could be done about this issue. PMP is really intended as some kind of memory interface, and it isn’t unreasonable for it to only maintain valid data for certain periods of time, modifying control signals to define this valid data period. That PMP can be used to drive LCD panels is merely a result of those panels themselves upholding this kind of interface. For those of you familiar with microcontrollers, the solution to my problem was probably obvious several paragraphs ago, but it needed me to reconsider my assumptions and requirements before I realised what I should have been doing all along.

Unlike SPI, which concerns itself with the bit-by-bit serial output of data, PMP concerns itself with the multiple-bits-at-once parallel output of data, and all I wanted to do was to present multiple bits to a memory location and have them translated to a collection of separate signals. But, of course, this is exactly how normal I/O (input/output) pins are provided on microcontrollers! They all seem to provide “PORT” registers whose bits correspond to output pins, and if you write a value to those registers, all the pins can be changed simultaneously. (This feature is obscured by platforms like Arduino where functions are offered to manipulate only a single pin at once.)

And so, I changed the DMA destination to be the PORTB register, which on the PIC32MX270 is the only PORT register with enough bits corresponding to I/O pins to be useful enough for this application. Even then, PORTB does not have a complete mapping from bits to pins: some pins that are available in other devices have been dedicated to specific functions on the PIC32MX270F256B and cannot be used for I/O. So, it turns out that we can only employ at most seven bits of our pixel data in generating signal data:

PORTB Pin Availability on the PIC32MX270F256B
Pins
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
RPB15 RPB14 RPB13 RPB11 RPB10 RPB9 RPB8 RPB7 RPB5 RPB4 RPB3 RPB2 RPB1 RPB0

We could target the first byte of PORTB (bits 0 to 7) or the second byte (bits 8 to 15), but either way we will encounter an unmapped bit. So, instead of choosing a colour representation making use of eight bits, we have to make do with only seven.

Initially, not noticing that RPB6 was not available, I was using a “RRRGGGBB” or “332″ representation. But persuaded by others in a similar predicament, I decided to choose a representation where each colour channel gets two bits, and then a separate intensity bit is used to adjust the final intensity of the basic colour result. This also means that greyscale output is possible because it is possible to balance the channels.

The 2-bit-per-channel plus intensity colours

The colours employing two bits per channel plus one intensity bit, perhaps not shown completely accurately due to circuit inadequacies and the usual white balance issues when taking photographs

It is worth noting at this point that since we have left the 8-bit limitations of the PMP peripheral far behind us now, we could choose to populate two bytes of PORTB at once, aiming for sixteen bits per pixel but actually getting fourteen bits per pixel once the unmapped bits have been taken into account. However, this would double our framebuffer memory requirements for the same resolution, and we don’t have that much memory. There may be devices with more than sixteen bits mapped in the 32-bit PORTB register (or in one of the other PORT registers), but they had better have more memory to be useful for greater colour depths.

Back in Black

One other matter presented itself as a problem. It is all very well generating a colour signal for the pixels in the framebuffer, but what happens at the end of each DMA transfer once a line of pixels has been transmitted? For the portions of the display not providing any colour information, the channel signals should be held at zero, yet it is likely that the last pixel on any given line is not at the lowest possible (black) level. And so the DMA transfer will have left a stray value in PORTB that could then confuse the monitor, producing streaks of colour in the border areas of the display, making the monitor unsure about the black level in the signal, and also potentially confusing some monitors about the validity of the picture, too.

As with the horizontal sync pulses, we need a prompt way of switching off colour information as soon as the pixel data has been transferred. We cannot really use an Output Compare unit because that only affects the value of a single output pin, and although we could wire up some kind of blanking in our external circuit, it is simpler to look for a quick solution within the capabilities of the microcontroller. Fortunately, such a quick solution exists: we can “chain” another DMA channel to the one providing the pixel data, thereby having this new channel perform a transfer as soon as the pixel data has been sent in its entirety. This new channel has one simple purpose: to transfer a single byte of black pixel data. By doing this, the monitor will see black in any borders and beyond the visible regions of the display.

Wiring Up

Of course, the microcontroller still has to be connected to the monitor somehow. First of all, we need a way of accessing the pins of a VGA socket or cable. One reasonable approach is to obtain something that acts as a socket and that breaks out the different signals from a cable, connecting the microcontroller to these broken-out signals.

Wanting to get something for this task quickly and relatively conveniently, I found a product at a local retailer that provides a “male” VGA connector and screw-adjustable terminals to break out the different pins. But since the VGA cable also has a male connector, I also needed to get a “gender changer” for VGA that acts as a “female” connector in both directions, thus accommodating the VGA cable and the male breakout board connector.

Wiring up to the broken-out VGA connector pins is mostly a matter of following diagrams and the pin numbering scheme, illustrated well enough in various resources (albeit with colour signal transposition errors in some resources). Pins 1, 2 and 3 need some special consideration for the red, green and blue signals, and we will look at them in a moment. However, pins 13 and 14 are the horizontal and vertical sync pins, respectively, and these can be connected directly to the PIC32 output pins in this case, since the 3.3V output from the microcontroller is supposedly compatible with the “TTL” levels. Pins 5 through 10 can be connected to ground.

We have seen mentions of colour signals with magnitudes of up to 0.7V, but no explicit mention of how they are formed has been presented in this article. Fortunately, everyone is willing to show how they converted their digital signals to an analogue output, with most of them electing to use a resistor network to combine each output pin within a channel to produce a hopefully suitable output voltage.

Here, with two bits per channel, I take the most significant bit for a channel and send it through a 470ohm resistor. Meanwhile, the least significant bit for the channel is sent through a 1000ohm resistor. Thus, the former contributes more to the magnitude of the signal than the latter. If we were only dealing with channel information, this would be as much as we need to do, but here we also employ an intensity bit whose job it is to boost the channels by a small amount, making sure not to allow the channels to pollute each other via this intensity sub-circuit. Here, I feed the intensity output through a 2200ohm resistor and then to each of the channel outputs via signal diodes.

VGA Output Circuit

The circuit showing connections relevant to VGA output (generic connections are not shown)

The Final Picture

I could probably go on and cover other aspects of the solution, but the fundamental aspects are probably dealt with sufficiently above to help others reproduce this experiment themselves. Populating memory with usable image data, at least in this solution, involves copying data to RAM, and I did experience problems with accessing RAM that are probably related to CPU initialisation (as covered in my previous article) and to synchronising the memory contents with what the CPU has written via its cache.

As for the actual picture data, the RGB-plus-intensity representation is not likely to be the format of most images these days. So, to prepare data for output, some image processing is needed. A while ago, I made a program to perform palette optimisation and dithering on images for the Acorn Electron, and I felt that it was going to be easier to adapt the dithering code than it was to figure out the necessary techniques required for software like ImageMagick or the Python Imaging Library. The pixel data is then converted to assembly language data definition statements and incorporated into my PIC32 program.

VGA output from a PIC32 microcontroller

VGA output from a PIC32 microcontroller, featuring a picture showing some Oslo architecture, with the PIC32MX270 being powered (and programmed) by the Arduino Duemilanove, and with the breadboards holding the necessary resistors and diodes to supply the VGA breakout and, beyond that, the cable to the monitor

To demonstrate control over the visible region, I deliberately adjusted the display frequencies so that the monitor would consider the signal to be carrying an image 800 pixels by 600 pixels at a refresh rate of 60Hz. Since my framebuffer is only 256 lines high, I double the lines to produce 512 lines for the display. It would seem that choosing a line rate to try and produce 512 lines has the monitor trying to show something compatible with the traditional 640×480 resolution and thus lines are lost off the screen. I suppose I could settle for 480 lines or aim for 300 lines instead, but I actually don’t mind having a border around the picture.

The on-screen menu showing the monitor's interpretation of the signal

The on-screen menu showing the monitor's interpretation of the signal

It is worth noting that I haven’t really mentioned a “pixel clock” or “dot clock” so far. As far as the display receiving the VGA signal is concerned, there is no pixel clock in that signal. And as far as we are concerned, the pixel clock is only important when deciding how quickly we can get our data into the signal, not in actually generating the signal. We can generate new colour values as slowly (or as quickly) as we might like, and the result will be wider (or narrower) pixels, but it shouldn’t make the actual signal invalid in any way.

Of course, it is important to consider how quickly we can generate pixels. Previously, I mentioned a 24MHz clock being used within the PIC32, and it is this clock that is used to drive peripherals and this clock’s frequency that will limit the transfer speed. As noted elsewhere, a pixel clock frequency of 25MHz is used to support the traditional VGA resolution of 640×480 at 60Hz. With the possibilities of running the “peripheral clock” in the PIC32MX270 considerably faster than this, it becomes a matter of experimentation as to how many pixels can be supported horizontally.

Some Oslo street art being displayed by the PIC32

Some Oslo street art being displayed by the PIC32

For my own purposes, I have at least reached the initial goal of generating a stable and usable video signal. Further work is likely to involve attempting to write routines to modify the framebuffer, maybe support things like scrolling and sprites, and even consider interfacing with other devices.

Naturally, this project is available as Free Software from its own repository. Maybe it will inspire or encourage you to pursue something similar, knowing that you absolutely do not need to be any kind of “expert” to stubbornly persist and to eventually get results!

The development of Global Scale

Free Software – Frank Karlitschek_ | 10:07, Monday, 22 May 2017

The architecture of Nextcloud is a classic Web Application architecture. I picked this architecture 7.5 years ago because it is very well known and is proven to be scaled relatively easily. This usually works with off the shelf technologies like http load balancers, clusters of Linux webservers and clustered databases.

But for many years users and customers asked for ways to distribute a single instance over several datacenters. A lot of users and customers run organizations that are not in one office or sometimes not even in one country or on one continent. So how can the service run distributed over different hosting centers on different continents?

Until now there was no good answer for this requirement.

Over the years I talked with users who experimented with different approaches. Unfortunately, they all didn’t work.

If you talk to storage people how to solve this challenge they say: No problem. Just use a distributed storage system like Gluster, Ceph, Hadoop or other.

If you talk to database people and how to do this they say: No problem! Just use one of the cluster and replication systems for Oracle, MySQL, MariaDB and others.

The challenge is how this all works together. If a user changes a file in Nextcloud then it is necessary that the file is changed, potential encryption keys are updated, log files are written, database tables are changed, email notification are sent, external workflow scripts are executed and a lot of other things happen. All these operations that are triggered by a single file change have to happen in an atomic way. It happens completely or not at all. Using database replication and storage replication independently will lead to a broken state and data loss. Additional problems are that you need the full bandwidth and storage in all data centers. So there is a lot of overhead.

The Global Scale architecture, that we designed, is currently the only solution, as far as I know, which solves this challenge.

Additional benefits are that the storage and database and overall operational costs decreases because simpler, commodity and more standard components can be used.

Another benefit is that the locality of data can be controlled. So if it is a legal requirement that certain files never leave a certain jurisdiction then this can be guaranteed with the right GS Balancer and File Access Control settings.

So far I only talk about the benefit that GS breaks a service down into several data centers. This is only half of the truth. Nextcloud Global Scale can be used in an even more radical way. You could stop using clustered Nextcloud instances in general. Killing all central storage boxes and databases and move completely to commodity hardware. Using only local storage and local databases and local caching. This changes the world completely and makes big storages, SAN, NFS and object store boxes completely obsolete.

The Global Scale architecture idea and implementation were developed over the last year with feedback and ideas from many different people. Storage experts, Database vendors, Linux distribution people, container and orchestration experts for easy automatic deployment and several big customers and users of Nextcloud.

The main inspiration came out of long discussions with the people at DeiC are running the National Research and Education Network of Denmark. DeiC was doing successful experiments with a similar architecture for a while already.

At Nextcloud we are committed to develop everything we do as open source. This includes this feature. Also, if you want to contribute to this architecture then no Contributer License Agreement is needed and you don’t need to transfer any rights to the Nextcloud company.

More information about Nextcloud Global Scale can be found here: https://nextcloud.com/globalscale

Saturday, 20 May 2017

GSoC: Second, third week of community bonding

vanitasvitae's blog » englisch | 21:56, Saturday, 20 May 2017

Hi all!

This is my report for the second, as well as the first half of the third week of GSoC community bonding, which I spent again working on finalizing my OMEMO code.

I dug deeper into writing test cases, mainly integration tests and I found quite a lot of small, undetectable bugs this way (Yay!). This strengthens my plan to work test driven during my GSoC project. The OMEMO code I currently work on was started as part of my bachelor thesis about 4 to 5 months ago and at this time, I was more concerned about having working code in the end, so I wrote no tests at all. Deploying test cases AFTER the code is already written is not only a tideous task, but its also often very difficult (because the code is not structured properly). So I learned my lesson the hard way :D

During testing I also found another bug in an XMPP server software, which prevents Smack from creating accounts on the server on the fly. Unfortunatelly this bug will not get fixed anymore for the version I use (installed from debian testing repository, which I thought was *reasonable* new), which keeps me from doing proper testing the way its meant to be done. I don’t have the time to compile the server software myselves. Instead, I work around this issue by creating the accounts manually everytime I run the test suite using a small bashscript.

I also had to deal with a really strange bug with file writing and reading. smack-omemo has a set of 4 integration tests, which all write data into a temporary directory. After each test, the directory is deleted to prevent tests influencing eachother. The issue was, that only the first test could read/write to the test directory. All subsequent tests failed for some reason. It took me a long time to notice, that there were two folders created (one in the working directory, another one in the subdirectory of the integration test framework). I am still not really sure what happened. The first folder was logged in all debug output, while files were written (by the first test) to the second filder. I guess it was caused by the temp directory being specified using a relative path, which messed up the tests, which were instanciated by the test framework using reflection. But I’m really not sure about this. Specifying the directory using an absolute path fixed the issue in the end.

Last but not least, me and Flow worked out some more details about my GSoC project (Implementing (encrypted) Jingle file transfer for Smack). The module will most likely be based upon java.nio to be scalable in the future. Flow also emphasized that the API should be as easy to use as possible, but at the same time powerful and extensible, which is a nice challenge (and probably a common one within the XMPP community). My initial plan was to create a XEP for OMEMO encrypted Jingle file transfer. We decided, that it would be of more value, to specify the XEP in a way, which allows arbitrary encryption techniques instead of being OMEMO exclusive.

Currently there is a little bit of tension in the community regarding the OMEMO specification. I really hope (and believe) there is a solution which is suitable of making everybody happy and I’m looking forward to participate in an open discussion :)

Happy hacking!

Benchmarking microSD cards and more

Daniel's FSFE blog | 00:00, Saturday, 20 May 2017

It’s worth re-visiting this blog-post as I am going to do more benchmarks and add more results over time.

Motivation

If you ever tried using a flash drive (such a SD, microSD, USB drive or CF) for your root or home filesystem on a small computing device or smartphone, you probably have noticed that flash cards are in most cases a lot slower than integreted eMMC flash. Since most filesystems use 4k blocks, the random write/read performance using 4k blocks is what matters most in such scenarios. And while flash cards don’t come close to internal flash in these disciplines, there are significant differences between the models.

Jeff Geerling [1,2] has already benchmarked the performance of various microSD cards on different models of the “blobby” Raspberry Pi. I had a number of different microSD cards at hand and I tried to replicate his results on my sample. In addition, I extended the comparison to regular SD cards and USB drives.

Disclaimer


All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

Environment and tools

I encourage every reader to replicate my benchmarks and share their results. Therefore, I first want to describe how the measures were taken.

General equipment

For the benchmarks, I used the following hardware on my workstation (listing only relevant parts):

  • Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
  • Logilink CR0034A USB 3.0 card reader [5], used for (m)SD cards
  • Apacer ? USB 2.0 card reader, used for CF cards
  • Debian 9 “Stretch”

In the past, I also did iozone benchmarks using a Logilink CR0015 USB 2.0 (m)SD card reader [4] which are provided at the end of the article for reference.

iozone Benchmarks

For the first tests, Just like Jeff, I used the open source (but non-free) tool “iozone” [3] in the current stable version that is available on Debian Stretch (3.429).

I disabled caches and benchmarked on raw devices to avoid measuring filesystem overhead. Therefore, I used the following call to iozone to run the benchmarks (/dev/sde is my sdcard):

$ iozone -e -I -a -s 100M -r 4k -r 16M -i 0 -i 1 -i 2 -f /dev/sdX

(replacing sdX by the actual device name)

ddrescue benchmarks

I used GNU ddrescue (1.21) in synchronous mode to overwrite a device with zeroes and measured the performance there. For this purpose, I used the following call to do a write benchmark:

$ ddrescue -D --force /dev/zero /dev/sdX

(replacing sdX by the actual device name)

Similarly, to measure the read performance I used (I had more than 100GB RAM left for tmpfs):

$ ddrescue -D --force /dev/sdX /tmp/foo

(replacing sdX by the actual device name)

Results

iozone

Here are the iozone results:

Manufacturer Make / model Type Speed Class Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s)
Samsung Evo+ (2017) mSDXC U3 64 GB 42.47 40.10 8.48 2.29
Samsung Evo+ mSDHC U1 32 GB 42,53 34,48 8.01 3.97
Sandisk Ultra mSDXC C10 64 GB 18.21 19.19 5.35 1.06
Sandisk Cruzer USB 2.0 4 GB 22.89 4.05 6.86 0.44
Kingston Data Traveller G3 USB 2.0 8 GB 21.52 4.96 6.96 0.08
Super Talent Pico C USB 2.0 8 GB 36.72 13.77 DNF DNF
Sandisk Ultra USB 3.0 32 GB 6.41 2.27 148.46 52.97
Samsung ? USB 2.0 1 GB 21.48 13.47 11.10 0.23
Transcend ? CF 80x 1 GB 20.19 12.56 7.56 0.19
Data Power ? (SLC) CF ? 1 GB 12.91 4.97 6.45 0.003
Toshiba ? CF ? 256 MB 7.52 2.86 3.78 0.003
Samsung Evo SDHC U1 32 GB 9.29 4.01 46.34 17.50
? mSDHC ? ? 8 GB tbd tbd tbd tbd

ddrescue

Here are the results using the USB 3.0 card reader or the device directly (for usb drives):

Manufacturer Make / model Type Rating Capacity Avg read (MB/s) Avg write (MB/s)
Sandisk Evo+ mSDHC U1 32 GB 41.95 21.54
Sandisk Cruzer USB 2.0 4 GB 36.37 2.64
Super Talent Pico C USB 2.0 8 GB 36.29 19.80
Sandisk Ultra USB 3.0 32 GB 115.00 24.62
Samsung ? USB 2.0 1 GB 21.63 14.62
Kingston Data Traveller G3 USB 3.0 8 GB 20.64 5.07
Transcend ? CF 80x 1 GB 19.26 12.60
Data Power ? (SLC) CF ? 1 GB 12.68 5.02
Toshiba ? CF ? 256 MB 7.33 2.82
Samsung Evo SDHC U1 32 GB 45,39 19.39
? mSDHC ? ? 8 GB 15.64 5.45

Legacy USB 2.0 iozone results

Previously, I ran the iozone benchmarks using the USB 2.0 card reader. They might be relevant, if you are looking for a storage device which you want to put in a computing device with a slow card reader:

Manufacturer Make / model Type Speed Class Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s)
Adata ? mSD C4 32 GB 9.58 2.97 2.62 0.64
Samsung Evo mSDHC U1 32 GB 17.80 10.13 4.45 1.32
Samsung Evo+ mSDHC U1 32 GB 18.10 14.27 5.28 2.86
Samsung Evo+ (2017) mSDXC U3 64 GB 16,98 8,03 7,99 2,99
Sandisk Extreme mSDXC U3 64 GB 18.44 11.22 5.20 2.36
Sandisk Ultra mSDHC C10 64 GB 18.16 8.69 3.73 0.80
Toshiba Exceria mSDXC U3 64 GB 16.10 6.60 3.82 0.09
Kingston ? mSD C4 8 GB 14.69 3.71 3.97 0.18

Verdict and future work

I can confirm Jeff’s results about microSD cards and would also recommend the Evo+ (both the old and the 2017 model) which have the best 4K random write performance of the sample. On the other hand, I am very disappointed about the Toshiba Exceria card. Actually running a device on this card with a very sluggish performance was the reason why I took this benchmark initiative. And indeed, after switching to the Evo+, the device feels much snappier now.

Of course, it would be worthwhile to add more cards and drives to this benchmark. Also, using fio instead of the non-free iozone might be interesting. Furthermore, doing the benchmarks internally on the device or using a other USB 3.0 card readers might be also interesting.

References

[1] http://www.pidramble.com/wiki/benchmarks/microsd-cards
[2] http://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-pi-microsd-card
[3] http://www.iozone.org/
[4] http://www.logilink.eu/Products_LogiLink/Notebook-Computer_Accessories/Card_Reader/Cardreader_USB_20_Stick_external_for_SD-MMC_CR0015.htm
[5] http://www.logilink.org/Produkte_LogiLink/Notebook-Computerzubehoer/Kartenleser/Cardreader_USB_30_SD-SD-HC-Micro_SD-Micro_SD-HC_CR0034A.htm?seticlanguage=en

Friday, 19 May 2017

Evaluating PIC32 for Hardware Experiments

Paul Boddie's Free Software-related blog » English | 21:45, Friday, 19 May 2017

Some time ago I became aware of the PIC32 microcontroller platform, perhaps while following various hardware projects, pursuing hardware-related interests, and looking for pertinent documentation. Although I had heard of PIC microcontrollers before, my impression of them was that they were mostly an alternative to other low-end computing products like the Atmel AVR series, but with a development ecosystem arguably more reliant on its vendor than the Atmel products for which tools like avr-gcc and avrdude exist as Free Software and have gone on to see extensive use, perhaps most famously in connection with the Arduino ecosystem.

What made PIC32 stand out when I first encountered it, however, was that it uses the MIPS32 instruction set architecture instead of its own specialised architecture. Consequently, instead of being reliant on the silicon vendor and random third-party tool providers for proprietary tools, the possibility of using widely-used Free Software tools exists. Moreover, with a degree of familiarity with MIPS assembly language, thanks to the Ben NanoNote, I felt that there might be an opportunity to apply some of my elementary skills to another MIPS-based system and gain some useful experience.

Some basic investigation occurred before I made any attempt to acquire hardware. As anyone having to pay attention to the details of hardware can surely attest, it isn’t much fun to obtain something only to find that some necessary tool required for the successful use of that hardware happens to be proprietary, only works on proprietary platforms, or is generally a nuisance in some way that makes you regret the purchase. Looking around at various resources such as the Pinguino Web site gave me some confidence that there were people out there using PIC32 devices with Free Software. (Of course, the eventual development scenario proved to be different from that envisaged in these initial investigations, but previous experience has taught me to expect that things may not go as originally foreseen.)

Some Discoveries

So that was perhaps more than enough of an introduction, given that I really want to focus on some of my discoveries in using my acquired PIC32 devices, hoping that writing them up will help others to use Free Software with this platform. So, below, I will present a few discoveries that may well, for all I know, be “obvious” to people embedded in the PIC universe since it began, or that might be “superfluous” to those happy that Microchip’s development tools can obscure the operation of the hardware to offer a simpler “experience”.

I should mention at this point that I chose to acquire PDIP-profile products for use with a solderless breadboard. This is the level of sophistication at which the Arduino products mostly operate and it allows convenient prototyping involving discrete components and other electronic devices. The evidence from the chipKIT and Pinguino sites suggested that it would be possible to set up a device on a breadboard, wire it up to a power supply with a few supporting components, and then be able to program it. (It turned out that programming involved another approach than indicated by that latter reference, however.)

The 28-pin product I elected to buy was the PIC32MX270F256B-50/SP. I also bought some capacitors that were recommended for wiring up the device. In the picture below, you can just about see the capacitor connecting two of the pins, and there is also a resistor pulling up one of the pins. I recommend obtaining a selection of resistors of different values so as to be able to wire up various circuits as you encounter them. Note that the picture does not show a definitive wiring guide: please refer to the product documentation or to appropriate project documentation for such things.

PIC32 device on a mini-breadboard

PIC32 device on a mini-breadboard connected to an Arduino Duemilanove for power and programming

Programming the Device

Despite the apparent suitability of a program called pic32prog, recommended by the “cheap DIY programmer” guide, I initially found success elsewhere. I suppose it didn’t help that the circuit diagram was rather hard to follow for me, as someone who isn’t really familiar with certain electrical constructs that have been mixed in, arguably without enough explanation.

Initial Recommendation: ArduPIC32

Instead, I looked for a solution that used an Arduino product (not something procured from ephemeral Chinese “auction site” vendors) and found ArduPIC32 living a quiet retirement in the Google Code Archive. Bypassing tricky voltage level conversion and connecting an Arduino Duemilanove with the PIC32 device on its 5V-tolerant JTAG-capable pins, ArduPIC32 mostly seemed to work, although I did alter it slightly to work the way I wanted to and to alleviate the usual oddness with Arduino serial-over-USB connections.

However, I didn’t continue to use ArduPIC. One reason is that programming using the JTAG interface is slow, but a much more significant reason is that the use of JTAG means that the pins on the PIC32 associated with JTAG cannot be used for other things. This is either something that “everybody” knows or just something that Microchip doesn’t feel is important enough to mention in convenient places in the product documentation. More on this topic below!

Final Recommendation: Pickle (and Nanu Nanu)

So, I did try and use pic32prog and the suggested circuit, but had no success. Then, searching around, I happened to find some very useful resources indeed: Pickle is a GPL-licensed tool for uploading data to different PIC devices including PIC32 devices; Nanu Nanu is a GPL-licensed program that runs on AVR devices and programs PIC32 devices using the ICSP interface. Compiling the latter for the Arduino and uploading it in the usual way (actually done by the Makefile), it can then run on the Arduino and be controlled by the Pickle tool.

Admittedly, I did have some problems with the programming circuit, most likely self-inflicted, but the developer of these tools was very responsive and, as I know myself from being in his position in other situations, provided the necessary encouragement that was perhaps most sorely lacking to get the PIC32 device talking. I used the “sink” or “open drain” circuit so that the Arduino would be driving the PIC32 device using a suitable voltage and not the Arduino’s native 5V. Conveniently, this is the default configuration for Nanu Nanu.

PIC32 on breadboard with Arduino programming circuit

PIC32 on breadboard with Arduino programming circuit (and some LEDs for diagnostic purposes)

I should point out that the Pinguino initiative promotes USB programming similar to that employed by the Arduino series of boards. Even though that should make programming very easy, it is still necessary to program the USB bootloader on the PIC32 device using another method in the first place. And for my experiments involving an integrated circuit on a breadboard, setting up a USB-based programming circuit is a distraction that would have complicated the familiarisation process and would have mostly duplicated the functionality that the Arduino can already provide, even though this two-stage programming circuit may seem a little contrived.

Compiling Code for the Device

This is perhaps the easiest problem to solve, strongly motivating my choice of PIC32 in the first place…

Recommendation: GNU Toolchain

PIC32 uses the MIPS32 instruction set. Since MIPS has been around for a very long time, and since the architecture was prominent in workstations, servers and even games consoles in the late 1980s and 1990s, remaining in widespread use in more constrained products such as routers as this century has progressed, the GNU toolchain (GCC, binutils) has had a long time to comfortably support MIPS. Although the computer you are using is not particularly likely to be MIPS-based, cross-compiling versions of these tools can be built to run on, say, x86 or x86-64 while generating MIPS32 executable programs.

And fortunately, Debian GNU/Linux provides the mipsel-linux-gnu variant of the toolchain packages (at least in the unstable version of Debian) that makes the task of building software simply a matter of changing the definitions for the compiler, linker and other tools in one’s Makefile to use these variants instead of the unprefixed “host” gcc, ld, and so on. You can therefore keep using the high-quality Free Software tools you already know. The binutils-mipsel-linux-gnu package provides an assembler, if you just want to practise your MIPS assembly language, as well as a linker and tools for creating binaries. Meanwhile, the gcc-mipsel-linux-gnu package provides a C compiler.

Perhaps the only drawback of using the GNU tools is that people using the proprietary tools supplied by Microchip and partners will post example code that uses special notation interpreted in a certain way by those products. Fortunately, with some awareness that this is going on, we can still support the necessary functionality in our own way, as described below.

Configuring the Device

With the PIC32, and presumably with the other PIC products, there is a distinct activity of configuring the device when programming it with program code and data. This isn’t so obvious until one reads the datasheets, tries to find a way of changing some behaviour of the device, and then stumbles across the DEVCFG registers. These registers cannot be set in a running program: instead, they are “programmed” before the code is run.

You might wonder what the distinction is between “programming” the device to take the code you have written and “programming” the configuration registers, and there isn’t much difference conceptually. All that matters is that you have your program written into one part of the device’s memory and you also ask for certain data values to be written to the configuration registers. How this is done in the Microchip universe is with something like this:

#pragma config SOMETHING = SOMEVALUE;

With a basic GNU tool configuration, we need to find the equivalent operation and express it in a different way (at least as far as I know, unless someone has implemented an option to support this kind of notation in the GNU tools). The mechanism for achieving this is related to the linker script and is described in a section of this article presented below. For now, we will concentrate on what configuration settings we need to change.

Recommendation: Disable JTAG

As briefly mentioned above, one thing that “everybody” knows, at least if they are using Microchip’s own tools and are busy copying code from Microchip’s examples, is that the JTAG functionality takes over various pins and won’t let you use them, even if you switch on a “peripheral” in the microcontroller that needs to use those pins. Maybe I am naive about how intrusive JTAG should or should not be, but the lesson from this matter is just to configure the device to not have JTAG features enabled and to use the ICSP programming method recommended above instead.

Recommendation: Disable Watchdog Timer

Again, although I am aware of the general concept of a watchdog timer – something that resets a device if it thinks that the device has hung, crashed, or experienced something particularly bad – I did not expect something like this to necessarily be configured by default. In case it is, and one does see lots of code assuming so, then it should be disabled as well. Otherwise, I can imagine that you might experience spontaneous resets for no obvious reason.

Recommendation: Configure the Oscillator and Clocks

If you need to change the oscillator frequency or the origin of the oscillator used by the PIC32, it is perhaps best to do this in the configuration registers rather than try and mess around with this while the device is running. Indeed, some configuration is probably unavoidable even if there is a need to, say, switch between oscillators at run-time. Meanwhile, the relationship between the system clock (used by the processor to execute instructions) and the peripheral clock (used to interact with other devices and to run things like timers) is defined in the configuration registers.

Linker Scripts

So, to undertake the matter of configuration, a way is needed to express the setting of configuration register values in a general way. For this, we need to take a closer look at linker scripts. If you routinely compile and link software, you will be using linker scripts without realising it, because such scripts are telling the tools things about where parts of the program should be stored, what kinds of addresses they should be using, and so on. Here, we need to take more of an interest in such matters.

Recommendation: Expand a Simple Script

Writing linker scripts does not seem like much fun. The syntax is awkward to read and to produce as a human being, and knowledge about tool output is assumed. However, it is possible to start with a simple script that works for someone else in a similar situation and to modify it conservatively in order to achieve what you need. I started out with one that just defined the memory regions and a few sections. To avoid reproducing all the details here, I will just show what the memory regions for a configuration register look like:

  config2                    : ORIGIN = 0xBFC00BF4, LENGTH = 0x4
  physical_config2           : ORIGIN = 0x3FC00BF4, LENGTH = 0x4

This will be written in the MEMORY construct in the script. What they tell us is that config2 is a region of four bytes starting at virtual address 0xBFC00BF4, which is the location of the DEVCFG2 register as specified in the PIC32MX270 documentation. However, this register actually has a physical address of 0x3FC00BF4. The difference between virtual addresses and physical addresses is perhaps easiest to summarise by saying that CPU instructions would use the virtual address when referencing the register, whereas the actual memory of the device employs the physical address to refer to this four-byte region.

Meanwhile, in the SECTIONS construct, there needs to be something like this:

  .devcfg2 : {
        *(.devcfg2)
        } > config2 AT > physical_config2

Now you might understand my remark about the syntax! Nevertheless, what these things do is to tell the tools to put things from a section called .devcfg2 in the physical_config2 memory region and, if there were to be any address references in the data (which there isn’t in this case), then they would use the addresses in the config2 region.

Recommendation: Define Configuration Sections and Values in the Code

Since I have been using assembly language, here is what I do in my actual program source file. Having looked at the documentation and figured out which configuration register I need to change, I introduce a section in the code that defines the register value. For DEVCFG2, it looks like this:

.section .devcfg2, "a"
.word 0xfff9fffb        /* DEVCFG2<18:16> = FPLLODIV<2:0> = 001;
                        DEVCFG2<6:4> = FPLLMUL<2:0> = 111;
                        DEVCFG2<2:0> = FPLLIDIV<2:0> = 011 */

Here, I fully acknowledge that this might not be the most optimal approach, but when you’re learning about all these things at once, you make progress as best you can. In any case, what this does is to tell the assembler to include the .devcfg2 section and to populate it with the specified “word”, which is four bytes on the 32-bit PIC32 platform. This word contains the value of the register which has been expressed in hexadecimal with the most significant digit first.

Returning to our checklist of configuration tasks, what we now need to do is to formulate each one in terms of configuration register values, introduce the necessary sections and values, and adjust the values to contain the appropriate bit patterns. The above example shows how the DEVCFG2 bits are adjusted and set in the value of the register. Here is a short amplification of the operation:

DEVCFG2 Bits
31…28 27…24 23…20 19…16 15…12 11…8 7…4 3…0
1111 1111 1111 1001
FPLLODIV<2:0>
1111 1111 1111
FPLLMUL<2:0>
1011
FPLLIDIV<2:0>
f f f 9 f f f b

Here, the underlined bits are those of interest and have been changed to the desired values. It turns out that we can set the other bits as 1 for the functionality we want (or don’t want) in this case.

By the way, the JTAG functionality is disabled in the DEVCFG0 register (JTAGEN, bit 2, on this device). The watchdog timer is disabled in DEVCFG1 (FWDTEN, bit 23, on this device).

Recommendation: Define Regions for Exceptions and Interrupts

The MIPS architecture has the processor jump to certain memory locations when bad things happen (exceptions) or when other things happen (interrupts). We will cover the details of this below, but while we are setting up the places in memory where things will reside, we might as well define where the code to handle exceptions and interrupts will be living:

  .flash : { *(.flash*) } > kseg0_program_mem AT > physical_program_mem

This will be written in the SECTIONS construct. It relies on a memory region being defined, which would appear in the MEMORY construct as follows:

  kseg0_program_mem    (rx)  : ORIGIN = 0x9D000000, LENGTH = 0x40000
  physical_program_mem (rx)  : ORIGIN = 0x1D000000, LENGTH = 0x40000

These definitions allow the .flash section to be placed at 0x9D00000 but actually be written to memory at 0x1D00000.

Initialising the Device

On the various systems I have used in the past, even when working in assembly language I never had to deal with the earliest stages of the CPU’s activity. However, on the MIPS systems I have used in more recent times, I have been confronted with the matter of installing code to handle system initialisation, and this does require some knowledge of what MIPS processors would like to do when something goes wrong or if some interrupt arrives and needs to be processed.

The convention with the PIC32 seems to be that programs are installed within the MIPS KSEG0 region (one of the four principal regions) of memory, specifically at address 0x9FC00000, and so in the linker script we have MEMORY definitions like this:

  kseg0_boot_mem       (rx)  : ORIGIN = 0x9FC00000, LENGTH = 0xBF0
  physical_boot_mem    (rx)  : ORIGIN = 0x1FC00000, LENGTH = 0xBF0

As you can see, this region is far shorter than the 512MB of the KSEG0 region in its entirety. Indeed, 0xBF0 is only 3056 bytes! So, we need to put more substantial amounts of code elsewhere, some of which will be tasked with handling things when they go wrong.

Recommendation: Define Exception and Interrupt Handlers

As we have seen, the matter of defining routines to handle errors and interrupt conditions falls on the unlucky Free Software developer in this case. When something goes wrong, like the CPU not liking an instruction it has just seen, it will jump to a predefined location and try and execute code to recover. By default, with the PIC32, this location will be at address 0×80000000 which is the start of RAM, but if the RAM has not been configured then the CPU will not achieve very much trying to load instructions from that address.

Now, it can be tempting to set the “exception base”, as it is known, to be the place where our “boot” code is installed (at 0x9FC00000). So if something bad does happen, our code will start running again “from the top”, a bit like what used to happen if you ever wrote BASIC code that said…

10 ON ERROR GOTO 10

Clearly, this isn’t particularly desirable because it can mask problems. Indeed, I found that because I had not observed a step in the little dance required to change interrupt locations, my program would be happily restarting itself over and over again upon receiving interrupts. This is where the work done in the linker script starts to pay off: we can move the exception handler, this being the code residing at the exception base, to another region of memory and tell the CPU to jump to that address instead. We should therefore never have unscheduled restarts occurring once this is done.

Again, I have been working in assembly language, so I position my exception handling code using a directive like this:

.section .flash, "a"

exception_handler:
    /* Exception handling code goes here. */

Note that .flash is what we mentioned in the linker script. After this directive, the exception handler is defined so that the CPU has something to jump to. But exceptions are just one kind of condition that may occur, and we also need to handle interrupts. Although we might be able to handle both things together, you can instead position an interrupt handler after the exception handler at a well-defined offset, and the CPU can be told to use that when it receives an interrupt. Here is what I do:

.org 0x200

interrupt_handler:
    /* Interrupt handling code goes here. *

The .org directive tells the assembler that the interrupt handler will reside at an offset of 0×200 from the start of the .flash section. This number isn’t something I have made up: it is defined by the MIPS architecture and will be observed by the CPU when suitably configured.

So that leaves the configuration. Although one sees a lot of advocacy for the “multi-vector” interrupt handling support of the PIC32, possibly because the Microchip example code seems to use it, I wanted to stick with the more widely available “single-vector” support which is what I have effectively described above: you have one place the CPU jumps to and it is then left to the code to identify the interrupt condition – what it is that happened, exactly – and then handle the condition appropriately. (Multi-vector handling has the CPU identify the kind of condition and then choose a condition-specific location to jump to all by itself.)

The following steps are required for this to work properly:

  1. Make sure that the CP0 (co-processor 0) STATUS register has the BEV bit set. Otherwise things will fail seemingly inexplicably.
  2. Set the CP0 EBASE (exception base) register to use the exception handler address. This changes the target of the jump that occurs when an exception or interrupt occurs.
  3. Set the CP0 INTCTL (interrupt control) register to use a non-zero vector spacing, even though this setting probably has no relevance to the single-vector mode.
  4. Make sure that the CP0 CAUSE (exception cause) register has the IV bit set. This tells the CPU that the interrupt handler is at that magic 0×200 offset from the exception handler, meaning that interrupts should be dispatched there instead of to the exception handler.
  5. Now make sure that the CP0 STATUS register has the BEV bit cleared, so that the CPU will now use the new handlers.

To enable exceptions and interrupts, the IE bit in the CP0 STATUS register must be set, but there are also other things that must be done for them to actually be delivered.

Recommendation: Handle the Initial Error Condition

As I found out with the Ben NanoNote, a MIPS device will happily run in its initial state, but it turns out that this state is some kind of “error” state that prevents exceptions and interrupts from being delivered, even though interrupts may have been enabled. This does make some kind of sense: if a program is in the process of setting up interrupt handlers, it doesn’t really want an interrupt to occur before that work is done.

So, the MIPS architecture defines some flags in the CP0 STATUS register to override normal behaviour as some kind of “fail-safe” controls. The ERL (error level) bit is set when the CPU starts up, preventing errors (or probably most errors, at least) as well as interrupts from interrupting the execution of the installed code. The EXL (exception level) bit may also be set, preventing exceptions and interrupts from occurring even when ERL is clear. Thus, both of these bits must be cleared for interrupts to be enabled and delivered, and what happens next rather depends on how successful we were in setting up those handlers!

Recommendation: Set up the Global Offset Table

Something that I first noticed when looking at code for the Ben NanoNote was the initialisation of a register to refer to something called the global offset table (or GOT). For anyone already familiar with MIPS assembly language or the way code is compiled for the architecture, programs follow a convention where they refer to objects via a table containing the addresses of those objects. For example:

Global Offset Table
Offset from Start Member
+0 0
+4 address of _start
+8 address of my_routine
+12 address of other_routine

But in order to refer to the table, something has to have the address of the table. This is where the $gp register comes in: it holds that address and lets the code access members of the table relative to the address.

What seems to work for setting up the $gp register is this:

        lui $gp, %hi(_GLOBAL_OFFSET_TABLE_)
        ori $gp, $gp, %lo(_GLOBAL_OFFSET_TABLE_)

Here, the upper 16 bits of the $gp register are set with the “high” 16 bits of the value assigned to the special _GLOBAL_OFFSET_TABLE_ symbol, which provides the address of the special .got section defined in the linker script. This value is then combined using a logical “or” operation with the “low” 16 bits of the symbol’s value.

(I’m sure that anyone reading this far will already know that MIPS instructions are fixed length and are the same length as the address being loaded here, so it isn’t possible to fit the address into the load instruction. So each half of the address has to be loaded separately by different instructions. If you look at the assembled output of an assembly language program employing the “li” instruction, you will see that the assembler breaks this instruction down into “lui” and “ori” if it needs to. Note well that you cannot rely on the “la” instruction to load the special symbol into $gp precisely because it relies on $gp having already been set up.)

Rounding Off

I could probably go on about MIPS initialisation rituals, setting up a stack for function calls, and so on, but that isn’t really what this article is about. My intention here is to leave enough clues and reminders for other people in a similar position and perhaps even my future self.

Even though I have some familiarity with the MIPS architecture, I suppose you might be wondering why I am not evaluating more open hardware platforms. I admit that it is partly laziness: I could get into doing FPGA-related stuff and deploy open microcontroller cores, maybe even combining them with different peripheral circuits, but that would be a longer project with a lot of familiarisation involved, plus I would have to choose carefully to get a product supported by Free Software. It is also cheapness: I could have ordered a HiFive1 board and started experimenting with the RISC-V architecture, but that board seems expensive for what it offers, at least once you disregard the pioneering aspects of the product and focus on the applications of interest.

So, for now, I intend to move slowly forwards and gain experiences with an existing platform. In my next article on this topic, I hope to present some of the things I have managed to achieve with the PIC32 and a selection of different components and technologies.

Containerising the FSFE, aiming for Kubernetes & 2020

free software - Bits of Freedom | 07:14, Friday, 19 May 2017

Containerising the FSFE, aiming for Kubernetes & 2020

When the FSFE started using virtualisation technologies on our servers, about 15 years ago, we decided to use Linux-Vservers, a budding project which Jacques Gélinas had started just a year or two prior. To those of us administering the FSFE machines at the time, it was quite like magic, being able to run several virtual containers on one machine, and being able to delegate control of them to different teams without fearing for the overall security or stability.

A number of years later, we started migrating our Linux-Vservers to Proxmox, and as of last week, we've also added Docker as an experiment for our deployments of new services. I will try to give a bit of an insight into what this looks like right this moment, with the caveat that this is a work in progress. I should also say up front that while we're currently using GitLab for some of our repositories, we also have a self-hosted installation of Gitea which isn't publicly released yet and part of the work we're doing is building towards actually being able to make better use of that Gitea, rather than GitLab.

Let's start at the bottom of the stack: the individual service. I'll be using our oidcp (OpenID Connect Provider) as an example in this.

Service level

The oidcp is the authentication provider of our new account management system. The master branch of this is on GitLab (https://gitlab.com/lukewm/oidcp/) and from the perspective of deployments, there are two critical pieces in that code, the Dockerfile and GitLab-CI File. The latter file is used by the tools for continuous integration and defines three separate stages: testing, building and deployment.

Changes to the master branch are typically made as merge requests. I maintain my own version of the branch for my own development, and when it's reached a certain point of maturity, I file a merge request against the master branch.

For any commit that is made, to a fork or branch of the main repository, the CI tools run through the three stages defined in the setup. This consists of quality checking the code, running our test suites and testing to build the code. Once all of those tasks have been done and succeeded, we get a nice green check mark showing that the commit has passed and -- if it's a merge request -- is probably fine to merge, from a quality and test perspective at least.

The Dockerfile is fairly simple, but if you haven't used Docker before, it may seem like magic. Think of Docker as a bunch of helper tools around a chroot. It's not entirely fair towards Docker, but it feels as if that's where it all started. Each Docker container has everything it needs to be a self-contained unit and doesn't (typically) share anything with any other container.

The Dockerfile for oidcp starts off by telling Docker that we would like our oidcp container to be built using Ubuntu zesty as the base container. This is how Docker typically work: you have a number of base containers, which you add something to, and then use those as new base containers for other work, and so on.

When constructing a container for oidcp, Docker starts by pulling in a complete Ubuntu zesty, copies over our oidcp code to that running system, sets up a few symlinks, and installs some Python packages needed. This creates an image which we can run on our host machine.

One final note on that Docker image, the Dockerfile also contain instructions about exposing port 8000 to the outside world. When a Docker image runs, it gets connected to a private network and doesn't typically expose any open ports to the outside world. In order to enable the outside world to reach our oidcp on port 8000, we need to explicitly expose it.

As we'll see later though, the "outside world" here is just the host machine. Exposing a port still doesn't mean that anyone outside of the host machine where the service run can access it. For this we need a bit more.

Running containers

Once we have an image built from a Dockerfile, we need to run it somewhere. In Düsseldorf, the FSFE has a cluster of three physical machines running Proxmox. On this cluster, I've created the virtual machine Lund, named after.. well, I'm honestly not sure right now. The naming scheme for the FSFE is that we name our virtual servers after chemists (Cavendish, Ekeberg, Auer, ...), and the physical machines after chemical elements (Argon, Zinc, Gallium, ...).

When I named Lund, I am absolutely sure I found a chemist with the last name "Lund", but when I now try to find said chemist, I come up empty. So even if there is one, he or she is probably not the most famous. But there you have it. One virtual machine. Lund.

Lund is installed with a basic Debian Jessie, and pretty much nothing else. We have an ansible playbook which cares for installing SSH keys, and I then added it to our backup cycle and installed Docker. Otherwise, there's nothing very specific set up on this machine.

Over on our (unofficial) FSFE Gitea installation, I have a repository with ansible playbooks for Lund. In this repository there's a playbook for each service we run on Lund, including one for the oidcp.

To install the oidcp on Lund, I run the corresponding playbook:

$ ansible-playbook -i hosts --ask-vault-pass oidcp.deploy.yml

This runs through the tasks in the oidcp playbook which in turn does the following:

  1. Build the image locally from a given Git repository. In principle, the image is already built on Gitlab, so this step should eventually be changed.
  2. Create a local network, which we'll need to allow selected running containers a way to communicate between each other. It's important to keep in mind that an ansible task is often described as the state one wishes the target object to be in. For the network, we say state: present, giving an indication to ansible we wish the network to be present. If it's not present, ansible will create it. If it's present, with the correct parameters, ansible won't change anything.
  3. We run a standard redis container. This one isn't actually built by us. We just make use of the distribution standard redis which is registered in the Docker registry with the name redis. So all we need to do is tell ansible we would like to run an instance of the redis image, connected to the network we've just created.
  4. We then run also our oidcp container, give it a bunch of environment variables to control its operation, and connect it to the same network, also indicating the alias "redis" should point to the running redis container. If we were to enter the oidcp container and run ping redis, this would work as expected.
  5. We similarly start a celery process using the same container and connected to the same network.

When we've made changes to our Git repository which I'd like to deploy, I just run the relevant ansible playbook and let ansible care for getting it on the machine.

But we're still missing one part: our oidcp provider exposed port 8000, but so far we only have a local network between some of our oidcp containers. Technically, we also have a local network which connects the host machine to its containers, so I can login to Lund and ping the IP numbers of the oidcp containers. But that network is also local to the host machine, so it's not reachable from the outside.

Connecting the network

We don't typically run our services on non-standard ports. We like to keep publicly accessible web services on port 80 (http), and even more on 443 (https). So we need something on Lund to answer on those ports, and we need to tie that to the relevant containers somehow.

I used to run this on the host machine itself, until I realised it's equally possible to put this in a Docker container too, and put the relevant configuration and deployment scripts into Git.

The way I've done this to allow for FSFE specific modifications is I've created a Git repository for fsfe-rp which contains the simplest Dockerfile one could ever imagine almost. It's entirety is:

FROM jwilder/nginx-proxy:alpine  
COPY sites-enabled/ /etc/nginx/conf.d/  

What this does is it creates a Docker image based on the already existing jwilder/nginx-proxy container (which in itself is based on the nginx container), of the alpine version. It then copies any FSFE local configuration files to the nginx configuration directory in the image we build.

Alpine Linux is a popular choice to base Docker containers on. It's a small and light weight distribution using musl libc (instead of GNU libc) and busybox (instead of GNU Bash etc). This makes the entire distribution tiny, around 5MB in size, which is perfect for Docker images where you don't need a fully fledged distribution. You just need the bare minimum to run a service.

What's particular about Jason Wilder's nginx-proxy container is that when it starts, it connects to the Docker daemon running on the host machine. It monitors the running Docker images and whenever a Docker image is started or removed, it springs into action.

It looks for all Docker images which are run with an environment variable VIRTUAL_HOST. For example, our oidcp installation has the following environment:

env:  
  VIRTUAL_HOST: id.fsfe.org

For any image running with such a variable, it automatically adds a configuration to the running nginx. For our oidcp, this configuration looks like this:

upstream id.fsfe.org {  
            ## Can be connect with "bridge" network
            # oidcp
            server 172.17.0.8:8000;
}
server {  
    server_name id.fsfe.org;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://id.fsfe.org;
    }
}

As you can see, it has automatically detected the running container for oidcp, added its domain name as the server name to listen for (on port 80), and added an upstream. The upstream is then connected to the local IP number of that container, with port 8000 (the one exposed by the Docker container).

Any request which comes in on port 80 with the domain name id.fsfe.org is automatically proxied to the oidcp container. This nginx container in itself has port 80 (and 443) exposed to the wider Internet, so requsts to Lund will first pass through the nginx container, and then down to whichever local container is relevant for the request.

Adding SSL

Of course it would be nice if we could also break free from only having HTTP and have HTTPS as the default. This is accomplished by Yves Blusseau's LetsEncrypt companion to nginx-proxy. This is a Docker container which, similarly to nginx-proxy, listens to Docker events and springs into action when a container is added or removed (as well as regularly every hour).

What it does is it looks at the running containers and their environment. It looks for an environment like this:

env:  
  VIRTUAL_HOST: id.fsfe.org
  LETSENCRYPT_HOST: id.fsfe.org
  LETSENCRYPT_EMAIL: jonas@fsfe.org

When it finds the LETSENCRYPT_HOST variable, it gets in touch with LetsEncrypt and requests an SSL certificate for that domain name. When it gets one, it installs it in a folder shared with nginx-proxy, and the configuration of nginx is updated so it also listens to, and responds to HTTPS requests, for the given host. It also updates the HTTP section, so any HTTP requests are automatically redirected to the HTTPS version.

As mention, it also runs every hour. It does this in order to also renew any certificates which may need it. Here's the log output from when it detected id.fsfe.org and generated a certificate for it, and the regular checks for renewals:

2017/05/12 12:44:44 Running '/app/update_certs'  
Creating/renewal id.fsfe.org certificates... (id.fsfe.org)  
2017-05-12 12:44:46,328:INFO:simp_le:1211: Generating new account key  
2017-05-12 12:44:47,178:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org  
2017-05-12 12:44:49,576:INFO:simp_le:1305: id.fsfe.org was successfully self-verified  
2017-05-12 12:44:49,825:INFO:simp_le:1313: Generating new certificate private key  
2017-05-12 12:44:52,338:INFO:simp_le:391: Saving key.pem  
2017-05-12 12:44:52,338:INFO:simp_le:391: Saving chain.pem  
2017-05-12 12:44:52,339:INFO:simp_le:391: Saving fullchain.pem  
2017-05-12 12:44:52,339:INFO:simp_le:391: Saving cert.pem  
...
Creating/renewal id.fsfe.org certificates... (id.fsfe.org)  
2017-05-18 19:36:12,356:INFO:simp_le:1383: Certificates already exist and renewal is not necessary, exiting with status code 1.  
Sleep for 3600s  
...

The end game & where to now?

So we're now at the point where any service added with the right environment will automatically have an SSL certificate generated for it, with a reverse proxy setup to accept requests on its behalf, forwarded to the local container where it runs.

There are a few parts which is missing from this setup. First of all, the deployment which is currently done with ansible should be automated as part of our continuous integration chain. When a Git commit is merged into our master branch, this should trigger the testing, building and deployment of the new version.

Ideally, we should also truly have a separate staging environment, so new versions first gets deployed in a staging environment where we can test it out before deploying it in production. There are also a lot of tools to make container orchestration easier and more manageable, including Kubernetes, which I could see us using in the future.

We still have some ways to go, and if you want to help us containerise the FSFE, we'd love your help!

Wednesday, 17 May 2017

Hacking the food chain in Switzerland

DanielPocock.com - fsfe | 18:41, Wednesday, 17 May 2017

A group has recently been formed on Meetup seeking to build a food computer in Zurich. The initial meeting is planned for 6:30pm on 20 June 2017 at ETH, (Zurich Centre/Zentrum, Rämistrasse 101).

The question of food security underlies many of the world's problems today. In wealthier nations, we are being called upon to trust a highly opaque supply chain and our choices are limited to those things that major supermarket chains are willing to stock. A huge transport and storage apparatus adds to the cost and CO2 emissions and detracts from the nutritional value of the produce that reaches our plates. In recent times, these problems have been highlighted by the horsemeat scandal, the Guacapocalypse and the British Hummus crisis.

One interesting initiative to create transparency and encourage diversity in our diets is the Open Agriculture (OpenAg) Initiative from MIT, summarised in this TED video from Caleb Harper. The food produced is healthier and fresher than anything you might find in a supermarket and has no exposure to pesticides.

An open source approach to food

An interesting aspect of this project is the promise of an open source approach. The project provides hardware plans, a a video of the build process, source code and the promise of sharing climate recipes (scripts) to replicate the climates of different regions, helping ensure it is always the season for your favour fruit or vegetable.

Do we need it?

Some people have commented on the cost of equipment and electricity. Carsten Agger recently blogged about permaculture as a cleaner alternative. While there are many places where people can take that approach, there are also many overpopulated regions and cities where it is not feasible. Some countries, like Japan, have an enormous population and previously productive farmland contaminated by industry, such as the Fukushima region. Growing our own food also has the potential to reduce food waste, as individual families and communities can grow what they need.

Whether it is essential or not, the food computer project also provides a powerful platform to educate people about food and climate issues and an exciting opportunity to take the free and open source philosophy into many more places in our local communities. The Zurich Meetup group has already received expressions of interest from a diverse group including professionals, researchers, students, hackers, sustainability activists and free software developers.

Next steps

People who want to form a group in their own region can look in the forum topic "Where are you building your Food Computer?" to find out if anybody has already expressed interest.

Which patterns from the free software world can help more people build more food computers? I've already suggested using Debian's live-wrapper to distribute a runnable ISO image that can boot from a USB stick, can you suggest other solutions like this?

Can you think of any free software events where you would like to see a talk or exhibit about this project? Please suggest them on the OpenAg forum.

There are many interesting resources about the food crisis, an interesting starting point is watching the documentary Food, Inc.

If you are in Switzerland, please consider attending the meeting on at 6:30pm on 20 June 2017 at ETH (Centre/Zentrum), Zurich.

One final thing to contemplate: if you are not hacking your own food supply, who is?

Tuesday, 16 May 2017

Building an antenna and receiving ham and shortwave stations with SDR

DanielPocock.com - fsfe | 18:34, Tuesday, 16 May 2017

In my previous blog on the topic of software defined radio (SDR), I provided a quickstart guide to using gqrx, GNU Radio and the RTL-SDR dongle to receive FM radio and the amateur 2 meter (VHF) band.

Using the same software configuration and the same RTL-SDR dongle, it is possible to add some extra components and receive ham radio and shortwave transmissions from around the world.

Here is the antenna setup from the successful SDR workshop at OSCAL'17 on 13 May:

After the workshop on Saturday, members of the OSCAL team successfully reconstructed the SDR and antenna at the Debian info booth on Sunday and a wide range of shortwave and ham signals were detected:

Here is a close-up look at the laptop, RTL-SDR dongle (above laptop), Ham-It-Up converter (above water bottle) and MFJ-971 ATU (on right):

Buying the parts

Component Purpose, Notes Price/link to source
RTL-SDR dongle Converts radio signals (RF) into digital signals for reception through the USB port. It is essential to buy the dongles for SDR with TCXO, the generic RTL dongles for TV reception are not stable enough for anything other than TV. ~ € 25
Enamelled copper wire, 25 meters or more Loop antenna. Thicker wire provides better reception and is more suitable for transmitting (if you have a license) but it is heavier. The antenna I've demonstrated at recent events uses 1mm thick wire. ~ € 10
4 (or more) ceramic egg insulators Attach the antenna to string or rope. Smaller insulators are better as they are lighter and less expensive. ~ € 10
4:1 balun The actual ratio of the balun depends on the shape of the loop (square, rectangle or triangle) and the point where you attach the balun (middle, corner, etc). You may want to buy more than one balun, for example, a 4:1 balun and also a 1:1 balun to try alternative configurations. Make sure it is waterproof, has hooks for attaching a string or rope and an SO-239 socket. from € 20
5 meter RG-58 coaxial cable with male PL-259 plugs on both ends If using more than 5 meters or if you want to use higher frequencies above 30MHz, use thicker, heavier and more expensive cables like RG-213. The cable must be 50 ohm. ~ € 10
Antenna Tuning Unit (ATU) I've been using the MFJ-971 for portable use and demos because of the weight. There are even lighter and cheaper alternatives if you only need to receive. ~ € 20 for receive only or second hand
PL-259 to SMA male pigtail, up to 50cm, RG58 Joins the ATU to the upconverter. Cable must be RG58 or another 50 ohm cable ~ € 5
Ham It Up v1.3 up-converter Mixes the HF signal with a signal from a local oscillator to create a new signal in the spectrum covered by the RTL-SDR dongle ~ € 40
SMA (male) to SMA (male) pigtail Join the up-converter to the RTL-SDR dongle ~ € 2
USB charger and USB type B cable Used for power to the up-converter. A spare USB mobile phone charge plug may be suitable. ~ € 5
String or rope For mounting the antenna. A ligher and cheaper string is better for portable use while a stronger and weather-resistent rope is better for a fixed installation. € 5

Building the antenna

There are numerous online calculators for measuring the amount of enamelled copper wire to cut.

For example, for a centre frequency of 14.2 MHz on the 20 meter amateur band, the antenna length is 21.336 meters.

Add an extra 24 cm (extra 12 cm on each end) for folding the wire through the hooks on the balun.

After cutting the wire, feed it through the egg insulators before attaching the wire to the balun.

Measure the extra 12 cm at each end of the wire and wrap some tape around there to make it easy to identify in future. Fold it, insert it into the hook on the balun and twist it around itself. Use between four to six twists.

Strip off approximately 0.5cm of the enamel on each end of the wire with a knife, sandpaper or some other tool.

Insert the exposed ends of the wire into the screw terminals and screw it firmly into place. Avoid turning the screw too tightly or it may break or snap the wire.

Insert string through the egg insulators and/or the middle hook on the balun and use the string to attach it to suitable support structures such as a building, posts or trees. Try to keep it at least two meters from any structure. Maximizing the surface area of the loop improves the performance: a circle is an ideal shape, but a square or 4:3 rectangle will work well too.

For optimal performance, if you imagine the loop is on a two-dimensional plane, the first couple of meters of feedline leaving the antenna should be on the plane too and at a right angle to the edge of the antenna.

Join all the other components together using the coaxial cables.

Configuring gqrx for the up-converter and shortwave signals

Inspect the up-converter carefully. Look for the crystal and find the frequency written on the side of it. The frequency written on the specification sheet or web site may be wrong so looking at the crystal itself is the best way to be certain. On my Ham It Up, I found a crystal with 125.000 written on it, this is 125 MHz.

Launch gqrx, go to the File menu and select I/O devices. Change the LNB LO value to match the crystal frequency on the up-converter, with a minus sign. For my Ham It Up, I use the LNB LO value -125.000000 MHz.

Click OK to close the I/O devices window.

On the Input Controls tab, make sure Hardware AGC is enabled.

On the Receiver options tab, change the Mode value. Commercial shortwave broadcasts use AM and amateur transmission use single sideband: by convention, LSB is used for signals below 10MHz and USB is used for signals above 10MHz. To start exploring the 20 meter amateur band around 14.2 MHz, for example, use USB.

In the top of the window, enter the frequency, for example, 14.200 000 MHz.

Now choose the FFT Settings tab and adjust the Freq zoom slider. Zoom until the width of the display is about 100 kHZ, for example, from 14.15 on the left to 14.25 on the right.

Click the Play icon at the top left to start receiving. You may hear white noise. If you hear nothing, check the computer's volume controls, move the Gain slider (bottom right) to the maximum position and then lower the Squelch value on the Receiver options tab until you hear the white noise or a transmission.

Adjust the Antenna Tuner knobs

Now that gqrx is running, it is time to adjust the knobs on the antenna tuner (ATU). Reception improves dramatically when it is tuned correctly. Exact instructions depend on the type of ATU you have purchased, here I present instructions for the MFJ-971 that I have been using.

Turn the TRANSMITTER and ANTENNA knobs to the 12 o'clock position and leave them like that. Turn the INDUCTANCE knob while looking at the signals in the gqrx window. When you find the best position, the signal strength displayed on the screen will appear to increase (the animated white line should appear to move upwards and maybe some peaks will appear in the line).

When you feel you have found the best position for the INDUCTANCE knob, leave it in that position and begin turning the ANTENNA knob clockwise looking for any increase in signal strength on the chart. When you feel that is correct, begin turning the TRANSMITTER knob.

Listening to a transmission

At this point, if you are lucky, some transmissions may be visible on the gqrx screen. They will appear as darker colours in the waterfall chart. Try clicking on one of them, the vertical red line will jump to that position. For a USB transmission, try to place the vertical red line at the left hand side of the signal. Try dragging the vertical red line or changing the frequency value at the top of the screen by 100 Hz at a time until the station is tuned as well as possible.

Try and listen to the transmission and identify the station. Commercial shortwave broadcasts will usually identify themselves from time to time. Amateur transmissions will usually include a callsign spoken in the phonetic alphabet. For example, if you hear "CQ, this is Victor Kilo 3 Tango Quebec Romeo" then the station is VK3TQR. You may want to note down the callsign, time, frequency and mode in your log book. You may also find information about the callsign in a search engine.

The video demonstrates reception of a transmission from another country, can you identify the station's callsign and find his location?

<video controls="" height="480" width="640">
<source src="https://video.danielpocock.com/oscal_demo3.mp4" type="video/mp4">
Your browser does not support the video tag.
</source></video>

If you have questions about this topic, please come and ask on the Debian Hams mailing list. The gqrx package is also available in Fedora and Ubuntu but it is known to crash on startup in Ubuntu 17.04. Users of other distributions may also want to try the Debian Ham Blend bootable ISO live image as a quick and easy way to get started.

Friday, 12 May 2017

Talk on nature and permaculture

agger's Free Software blog | 19:26, Friday, 12 May 2017

STATE OF THE DANISH NATURE (AND SOCIETY)

Talk given at the technoshamanism encounter in Aarhus, November 22, 2014.

I wish to talk about the Danish nature, and I’d like to start by showing
you some pictures. The first pictures I’m going to show you were taken
by the writer and photogropher Rune Engelbreth Larsen who has been
working a lot on documenting the Danish nature as it is today.

Photo (c) Rune Engelbreth LarsenWhat you se here is of course a Danish forest with a lake, and if you
look at these pictures (which Rune Engelbreth has been dedicating
several years of his life to taking, making excursions and going to the
most remote corners of Danish nature) you’ll find that
many places in Denmark we actually have a rich nature
with very beautiful places, e.g. like this.

Photo (c) Rune Engelbreth LarsenSo if you visit a place like this and many others, you’d be forgiven for
thinking that if there are so many wonderful places here in Denmark
where you can go to meditate and enjoy the nature, we must have a
beautiful and very diverse nature in this country. And it’s true -
you can indeed find places where you can
take such pictures. You can find many such places, also e.g. like this:

Photo (c) Rune Engelbreth LarsenThese examples of Danish nature are places with a really peaceful
atmosphere. They are very green and very nice, usually with very
comfortable temperatures. If you visit some of the best places in Danish
nature, you’ll find that these are places that you can really feel a
part of, where you can relax and be happy and feel a strong sense of
belonging. As a Danish person I grew up with this kind of nature, and
I really feel at home in these tranquil spots. This is a landscape to
which I can really connect, spiritually.

Photo (c) Rune Engelbreth Larsen

In the photo above, you see one of our great oak forests, in Langaa.
Originally, Danish forests consisted mainly of oak trees; today, species
like beech, birch and fir are much more common. We don’t really
know why this forest has been conserved as oak, but in pre-Christian
times oak trees were connected to the ancient Nordic religion – the oak
is the tree of Thor. Thus, a very old oak forest like this one can also
help us connect to that part of our ancestral history.

However, if you look at other parts of the Danish landscape, a very
different picture emerges. The following pictures were all taken during a
walk that I took one day several years ago when I was living in
Solbjerg, a small town south of Aarhus. I walked around the lake called
Solbjerg Sø in one end and Stilling Sø in the other. There are, as you
can see, nice pastures along its banks that you can follow.

The trees that you see in the background are a small forest that’s also
quite beautiful once you get inside.

From there, you can go up a hill road and get a good view of the
landscape (click to see the full panorama image).

You have the hillside, the trees in the background, the
lake, and it’s all very nice, but: as pleasant as all of this might
appear, these pictures are actually lying. The first pictures I chose to
illustrate this article are truthful enough – they really do show some
very nice spots of Danish nature. But the pictures from my walk are
lying, all of them. Solbjerg Sø is situated in a completely agricultural
district, and the only reason there are green pastures at its banks is
that it’s too humid, so they can’t get their tractors all the way down
there. The trees at their edge form a hedge, and the corn fields start
right on the other side. In the autumn and winter, the pastures are
flooded and it’s nearly impossible to walk there. This is partly because
the lake’s water level is rising slightly, but mostly it’s because of
the drainage pipes that carry water from the fields on the hillside,
making them more tractor- and corn-friendly. This drainage water will
also carry excess fertilizer and pesticides which are sprayed on the
crop, all of which are flowing directly onto the pastures on their way
to the lake.

If you enter the forest on the other side of the lake – as I did that
day, since I walked all the way around the lake as close to the banks as
possible – you’ll se that it has been cut back and replaced with corn
fields as far down the slope as possible, right down to the point where
the ground becomes so humid that no matter what they do and how many
draining pipes they put in the ground, they still won’t be able to till
it and thus they can’t grow corn on it. The hillside is drained as far
down as possible, and the water from the fields is conducted to that
wood and makes it much too humid, creating a very bad climate in the
wood. The water from the fields is polluted with nutrients from the
chemical fertilizers used for the corn. This means that the wood by the
lake is overgrown with nettles, and otherwise the biological diversity
is nil. The high humidity does make it an ideal breeding place for
mosquitoes, though. As idyllic as that wood may look at a distance, it
really isn’t a nice place.

The area around Solbjerg Sø is a good example of a very idyllic but
completely destroyed Danish landscape. And this is the real state of
most of Danish nature these days.

On the other hand, if you look at the picture below, you’ll see a
typical Danish landscape. If you look at an aerial photo, you’ll find
that this is what Denmark is: Corn fields. You’ll see some tiny patches
of green dispersed in the landscape. These are forests that they haven’t
felled yet and some very few bogs that they were unable to drain
enough to sow corn.

These landscapes are really dreary. What few people know is that Denmark
is the most heavily farmed country in Europe and one of the most heavily
farmed – and most destroyed – countries in the world.

My assessment of the situation is that, financially speaking, we’re
doing all of this for nothing. The most important crop on Danish corn
fields is barley, which is mainly grown in order to feed pigs and other
livestock. The real reason for having all these corn fields is that we
have to feed the pigs in the industrial pig farms so that we can export a
lot of pork meat.

Meanwhile, in these pig factories the pigs are being overmedicated. All
of them are usually treated preventively with antibiotics. This is
creating new antibiotic-resistant bacteria which may also infect humans.
Right now, there’s a bacterium originating in pig farms called MRSA
(Methicillin-resistant Staphylococcus Aureus) which is actually killing
people. In Denmark, four people died of MRSA infection from 2012 to
2014. In order to protect the pig farmers from “harm” (e.g., in the
form of consumer boykot), the government is keeping the outbreak
locations secret. This means that school classes may and do visit farms
with outbreaks of this deadly disease. The teachers are not allowed to
make a informed decision not to visit an infected farm, since that
information is confidential.

So this is the dilemma: Our agricultural production is destroying Danish
nature and undermining the health of future generations by creating
penicillin-resistant strains of bacteria which are already lethal in
some cases. At the same time, we don’t really have to produce food in
that way, because we don’t need to use all this space on corn fields in
order to feed ourselves. Most of our agricultural production is
exported. Of course, one might claim that we produce all this grain and
pork meat for sound economic reasons. The agriculture lobby is always
emphasizing this claim.

But the fact is that agriculture as practised in Denmark is not
financially sustainable. The only reason that Danish farmers survive
economically is that they’re heavily subsidized. If it weren’t for
subsidies, they would quickly go bankrupt. In other words, the present
overfarming is pointless. We’re sacrificing our country’s nature and
biological diversity to the very short-sighted interests of one single
profession whose productions methods are not even economically viable.

This is the first of the dilemmas I want to address: We’re destroying
our country in order to produce far more food than we need, and society
as a whole is losing money because of it. Meanwhile, there are many
environmentally and economically sustainable ways to produce food.

A related dilemma is that today’s consumer society is moving faster and
faster. This means that we as people are constantly pushed harder and
harder to perform. You have to do well in school, you have to get good
grades in secondary school, you have to work hard at the university, you
have to get a good exam. Once you get out you have to get a good job.
You’ll often have to be there at nine o’clock in the morning and stay
until five. In most jobs, you must be there during these hours or
you’ll get fired. If you get fired and you’re eligible for unemployment
benefit, society has all sorts of rules so you have to go to meetings,
attend “courses” and turn up at eight or nine o’clock in the morning
anyway. Only now you won’t get paid more than you need to barely keep
you from starving, so you better find a job soon! In other words, our
society isn’t free at all: We’re forced to become parts of consumer
society and we’re not actually free to choose not to be a part of it.

One way in which we could become more free to choose how to live would be
to become less dependent on money.

I’d like to propose that the two problems I’ve described right now – the
destruction of nature by unsustainable and unnecessary food production
and modern society’s endemic lack of personal freedom – have one single
solution. We, as individuals, can obtain the means to sustain ourselves
and each other without having to work for money in the way we do now.
It is theoretically possible for us to create a sustainable life and a
sustainable production of food and other necessities in such a way that
our society is and remains in balance with nature.

If we were to organize our food production according to the principles
of permaculture; that is, if we were to produce what we need in forest
gardens and carefully designed multicultures, we could avoid the use of
chemical fertilizers, pesticides and fossil fuels. In permaculture you
normally design your crops to balance and suplement each other, so you
don’t need to supply fertilizer – a careful selection of edible
perennials can help provide the nutrients they need and keep each
other’s pests and diseases away. The only monoculture crop we really
have to cultivate is wheat, since people do want bread. In order to
produce bread, we need about 500 square meters (organically farmed) for
each person, and that’s nothing compared to the area we’re currently
using for agriculture. Apart from that, we could organize our
agriculture entirely according to permaculture principles. We could
easily do without agriculture as an export industry, since it’s already
not actually making us any money, since it’s only profitable because
it’s subsidized. Giving up on the pig production, we could shrink the
agricultural land to a fraction of what it is today. Instead of using
60% of the country’s area for agriculture, we could feed ourselves with
just 10% of the space; and we could re-establish nature in the rest of
it.

That’s what we could do as a society if we want to transition to
something more sustainable. But we can also do something as individuals.
If we have the option we might, of course, shoot our proverbial uncle
in America and inherit all his millions – but most of us don’t have that
option. On the other hand, if we were to get cheap access to a garden -
I was lucky, as I was able to buy a house cheaply and I’m now building a
permaculture project there – we could put in a modest amount of work for
some years and afterwards we’d be able to get most of the food that we
need from that garden. It’s estimated that if you want to be
self-sufficient with food, you could get by with 400 square meters per
person, farmed very intensively. When I took my Permculture Design
Certificate, our teacher said that he wouldn’t try to do that himself -
he’d probably go for 800. But that’s still a moderately sized garden.
People could form communities with shared gardens – share the access,
share the cost, share the work, share the food – and you could have
communities of people who sustain themselves without ever spending any
money on food. So this transition is not just something we could do as
a society – we could do it ourselves and become vastly more independent
of having a job and being a part of consumer society. I’m not saying
this is the whole solution – there’s still some dependencies like how to
get the land in the first place – but it could be an important start for
many of us.

If you’re consuming pigs which have been produced by industrial
agriculture, then you’re also paying to have their fodder moved around,
you’re paying to have your fertilizers transported from abroad, you’re
paying to sustain a lot of carbon emissions. If you build a forest
garden in your own garden or help build a forest garden on a shared area
so that afterwards you can get most of your food from that forest
garden, you’re actually not emitting any carbon, you’re storing it. A
growing forest garden is actively removing carbon from the atmosphere
and storing it in the trees, in the roots beneath the ground and in all
the plants constantly growing in that garden. Calculations have been
made which show that if – and that’s a very big IF – we were to
transition to get a large part of our food from forest gardens and
sourcing that produce locally instead of having everything driven and
flown around, we would be able to take enough carbon out of the Earth’s
atmosphere to make a difference regardint the climate changes. I’m not
saying that this is going to happen because obviously it won’t, but we
could theoretically avert some of the dangers of climate change that
way.

Maybe we can’t save the world, but we can make ourselves more
independent and our lives freer and more fulfilling – and in doing so we
can at least try to contribute as little as possible to our own
destruction.

Thank you to the OSCAL team

DanielPocock.com - fsfe | 13:26, Friday, 12 May 2017

The welcome gift deserves its own blog post. If you want to know what is inside, I hope to see you at OSCAL'17.

Kamailio World and FSFE team visit, Tirana arrival

DanielPocock.com - fsfe | 09:48, Friday, 12 May 2017

This week I've been thrilled to be in Berlin for Kamailio World 2017, one of the highlights of the SIP, VoIP and telephony enthusiast's calendar. It is an event that reaches far beyond Kamailio and is well attended by leaders of many of the well known free software projects in this space.

HOMER 6 is coming

Alexandr Dubovikov gave me a sneak peek of the new version of the HOMER SIP capture framework for gathering, storing and analyzing messages in a SIP network.

exploring HOMER 6 with Alexandr Dubovikov at Kamailio World 2017

Visiting the FSFE team in Berlin

Having recently joined the FSFE's General Assembly as the fellowship representative, I've been keen to get to know more about the organization. My visit to the FSFE office involved a wide-ranging discussion with Erik Albers about the fellowship program and FSFE in general.

discussing the Fellowship program with Erik Albers

Steak and SDR night

After a hard day of SIP hacking and a long afternoon at Kamailio World's open bar, a developer needs a decent meal and something previously unseen to hack on. A group of us settled at Escados, Alexanderplatz where my SDR kit emerged from my bag and other Debian users found out how easy it is to apt install the packages, attach the dongle and explore the radio spectrum.

playing with SDR after dinner

Next stop OSCAL'17, Tirana

Having left Berlin, I'm now in Tirana, Albania where I'll give an SDR workshop and Free-RTC talk at OSCAL'17. The weather forecast is between 26 - 28 degrees celsius, the food is great and the weekend's schedule is full of interesting talks and workshops. The organizing team have already made me feel very welcome here, meeting me at the airport and leaving a very generous basket of gifts in my hotel room. OSCAL has emerged as a significant annual event in the free software world and if it's too late for you to come this year, don't miss it in 2018.

OSCAL'17 banner

Thursday, 11 May 2017

time offset and nagios nrpe check

Evaggelos Balaskas - System Engineer | 17:18, Thursday, 11 May 2017

What is the time?

Time offset is the amount of time that is off (or drift) from a specific value. In Linux systems, date is been calculating from the beginning of time. That is 00:00:00 1 January 1970 or as it called Unix Time and systems define date (time) as the number of seconds that have elapsed from 01.01.1970.

It is so important that even a few seconds off can cause tremendous disaster in data centers and applications.

Network Time

To avoid problems with time, systems must and should synchronize their time over the Internet every now and then. This is being done by asking a central NTP server via Network Time Protocol. The most common scenario for infrastructures is to have one or two NTP servers and then all the systems inside this infrastructure can synchronize their time from those machines.

Nagios - NRPE

In my case, I have a centralized NTP Daemon that runs on the Nagios Linux machine. That gives me the opportunity to check the EPOCH time of any system in my infrastructure against the time that the Nagios Server has.

Nagios Check

This is the script I am using:

# ebal, Thu, 11 May 2017 12:08:50 +0000

# EPOCH
TIME=$1
WARN=5
CRIT=10

# seconds
OFFSET=$( echo $(( $(date -d 'now ' +%s) - ${TIME} )) | sed -e 's#-##g' )

if [ "${OFFSET}" -lt "${WARN}" ]; then
        echo "OK"
        exit 0
elif [ "${OFFSET}" -ge "${CRIT}" ]; then
        echo "CRITICAL- ${OFFSET}"
        exit 2
elif [ "${OFFSET}" -lt "${CRIT}" ]; then
        echo "WARNING- ${OFFSET}"
        exit 1
else
        echo "UNKNOWN- ${OFFSET}"
        exit 3
fi

In a nutshell the script gets as the first argument an epoch time and calculate the diff between it’s own epoch time and that.

Example

./check_time_offset $(date -d 'now + 1 min' +%s)

The output is this:

CRITICAL- 60

Nrpe Configuration

This is the configuration for nrpe to run the check_time_offset

# tail -1 /etc/nrpe.d/time_offset.cfg

command[check_time_offset]=/usr/lib64/nagios/plugins/check_time_offset $ARG1$

Nagios Configuration

and this is my nagios configuration setup to use a remote nrpe :

define service{
        use                             service-critical
        hostgroup_name                  lnxserver01
        service_description             time_offset
        check_command                   check_nrpe!check_time_offset!$TIMET$
        }

Take a minute to observer a little better the nrpe command.

check_nrpe!check_time_offset!$TIMET$

TIMET

I was having problems passing the nagios epoch time as an argument on the definition of the above service.

Testing the nrpe command as below, I was getting the results I was looking for:

./check_nrpe -H lnxserver01 -c check_time_offset -a $(date -d 'now + 6 sec' +%s)

But is there a way to pass as a nagios argument the output of a command ?

  • No

A dear colleague of mine mentioned nagios macros:

Standard Macros in Nagios

$TIMET$     Current time stamp in time_t format (seconds since the UNIX epoch)

Perfect !!!

Tag(s): nagios, nrpe, ntp, time

How my car insurance exposed my position

Andrea Scarpino's blog | 00:00, Thursday, 11 May 2017

As many car insurances companies do, my car insurance company provides a satellite device that can be put inside your car to provide its location at any time in any place.

By installing such device in your car, the car insurance profiles your conduct, of course, but it could also help the police in finding your car if it gets stolen and you will probably get a nice discount over the insurance price (even up to 40%!). Long story short: I got one.

Often such companies also provide an “App” for smartphones to easily track your car when you are away or to monitor your partner…mine (the company!) does.

Then I downloaded my company’s application for Android, but unluckily it needs the Google Play Services to run. I am a FOSS evangelist and, as such, I try to use FOSS apps only and without gapps.

Luckily I’m also a developer and, as such, I try to develop the applications I need most; using mitmproxy, I started to analyze the APIs used by the App to write my own client.

Authentication

As soon as the App starts you need to authenticate yourself to enable the buttons that allow you to track your car. Fair enough.

The authentication form first asks for your taxpayer’s code; I put mine and under the hood it performs the following request:

curl -X POST -d 'BLUCS§<taxpayers_code>§-1' http://<domain>/BICServices/BICService.svc/restpostcheckpicf<company>

The Web service replies with a cell phone number (WTF?):

2§<international_calling_code>§<cell_phone_number>§-1

Wait. What do we already see here? Yes, besides the ugliest formatting ever and the fact the request uses plain HTTP, it takes only 3 arguments to get a cell phone number? And guess what? The first one and the latter are two constants. In fact, if we put an inexistent taxpayer’s code, by keeping the same values, we get:

-1§<international_calling_code>§§-100%

…otherwise we get a cell phone number for the given taxpayer’s code!

I hit my head and I continued the authentication flow.

After that, the App asks me to confirm the cell phone number it got is still valid, but it also wants the password I got via mail when subscribing the car insurance; OK let’s proceed:

curl -X POST -d 'BLUCS§<taxpayers_code>§<device_imei>§<android_id>§<device_brand>-<device_model>_unknown-<api_platform>-<os_version>-<device_code>§<cell_phone_number>§2§<password>§§-1' http://<domain>/BICServices/BICService.svc/restpostsmartphoneactivation<company>

The Web service responds with:

0§<some_code>§<my_full_name>

The some_code parameter changes everytime, so it seems to work as a “registration id”, but after this step the App unlocked the button to track my car.

I was already astonished at this point: how the authentication will work? Does it need this some_code in combination with my password at reach request? Or maybe it will ask for my taxpayer code?

Car tracking

I start implementing the car tracking feature, which allows to retrieve the last 20 positions of your car, so let’s analyze the request made by the App:

curl -X POST -d 'ASS_NEW§<car_license>§2§-1' http://<domain>/BICServices/BICService.svc/restpostlastnpositions<company>

The Web service responds with:

0§20§<another_code>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>§DD/MM/YYYY HH:mm:SS#<latitude>#<longitude>#0#1#1#1-<country>-<state>-<city>-<street>

WTH?!? No header?!? No cookie?!? No authentication parameters?!?

Yes, your assumption is right: you just need a car license and you get its last 20 positions. And what’s that another_code? I just write it down for the moment.

It couldn’t be real, I first thought (hoped) they stored my IP somewhere so I’m authorized to get this data now, so let’s try from a VPN…oh damn, it worked.

Then I tried with an inexistent car license and I got:

-2§TARGA NON ASSOCIATA%

which means: “that car license is not in our database”.

So what we could get here with the help of crunch? Easy enough: a list of car licenses that are covered by this company and last 20 positions for each one.

I couldn’t stop now.

The Web client

This car insurance company also provides a Web client which permits more operations, so I logged into to analyze its requests and while it’s hosted on a different domain, and it also uses a cookie for almost any request, it performs one single request to the domain I previously used. Which isn’t authenticated and got my attention:

curl http://<domain>/<company>/(S(<uuid>))/NewRemoteAuthentication.aspx?RUOLO=CL&ID=<another_code>&TARGA=<car_license>&CONTRATTO=<foo>&VOUCHER=<bar>

This one replies with an HTML page that is shown in the Web client:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" >
<HTML>
<HEAD>
    <title>NewRemoteAuthentication</title>
    <meta name="GENERATOR" Content="Microsoft Visual Studio .NET 7.1" />
    <meta name="CODE_LANGUAGE" Content="C#" />
    <meta name="vs_defaultClientScript" content="JavaScript"/>
    <meta name="vs_targetSchema" content="http://schemas.microsoft.com/intellisense/ie7" />
        <!--<meta content="IE=EmulateIE10" name="ie_compatibility" http-equiv="X-UA-Compatible" />-->
        <meta name="ie_compatibility" http-equiv="X-UA-Compatible" content="IE=7, IE=8, IE=EmulateIE9, IE=10, IE=11" />
</HEAD>
    <body>
    <form name="Form1" method="post" action="/<company>/(S(<uuid>))/NewRemoteAuthentication.aspx?RUOLO=CL&amp;ID=<another_code>&amp;TARGA=<car_license>" id="Form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwULLTIwNzEwODIsJFNAgEPKAJDIeBsdSpc2libGVnZGRic5McHC9+DqRx0H+jRt5O+/PLtw==" />

            <iframe id="frm1" src="NewRicerca.aspx" width="100%" height="100%"></iframe>


<SCRIPT language="JavaScript">
<!--
self.close
// -->
</SCRIPT>
</form>
</body>
</HTML>

It includes an iframe (sigh!), but that’s the interesting part!!! Look:

Car history

From that page you get:

  • the full name of the person that has subscribed the insurance;
  • the car model and brand;
  • the total amount of kilometers made by the car;
  • the total amount of travels (meant as “car is moving”) made by the car;
  • access to months travels details (how many travels);
  • access to day travels details (latitude, longitude, date and time);
  • access to months statistics (how often you use your car).

Car month history Car day history Car month_statistics

There are a lot of informations here and these statistics are available since the installation of the satellite device.

The request isn’t authenticated so I just have to understand the parameters to fill in. Often not all parameters are required and then I tried by removing someone to find out which are really needed. It turns out that I can simplify that as:

curl http://<domain>/<company>/(S(<uuid>))/NewRemoteAuthentication.aspx?RUOLO=CL&ID=<another_code>&TARGA=<car_license>

But there’s still a another_code there…mmm, wait it looks like the number I took down previously! And yes, it’s!

So, http://<domain>/<company>/(S(<uuid>))/NewRicerca.aspx is the page that really shows all the informations, but how do I generate that uuid thing?

I tried by removing it first and then I got an empty page. Sure, makes sense, how that page will ever know which data I’m looking for?

Then it must be the NewRemoteAuthentication.aspx page that does something; I tried again by removing the uuid from that url and to my full surprise it redirected me to the same url, but it also filled the uuid part as path parameter! Now I can finally invoke the NewRicerca.aspx using that uuid and read all the data!

Conclusion

You just need a car license which is covered by this company to get all the travels made by that car, the full name of the person owning it and its position in real time.

I reported this privacy flaw to the CERT Nazionale which wrote to the company.

The company fixed the leak 3 weeks later by providing new Web services endpoints that use authenticated calls. The company mailed its users saying them to update their App as soon as possible. The old Web services have been shutdown after 1 month and half since my first contact with the CERT Nazionale.

I got no bounty.

The company is a leading provider of telematics solutions.

Wednesday, 10 May 2017

GSoC: First week of community bonding

vanitasvitae's blog » englisch | 23:19, Wednesday, 10 May 2017

The first week of community bonding is nearly over and already it’s quite an experience for me. Me and Alameyo were very nicely welcomed by members of the igniterealtime project which really took care of making us able to jump right into the project.

I spent most of the time making myself more familiar with the Smack codebase by working on my OMEMO pull request, which comes closer and closer to a mergeable state. Currently I’m working together with my mentor Flo to make the API of the module as easy to use as possible. I learned the hard way, that singletons and generics are a pain when used together, but in the end it worked out quite well I’d say, although there is still some work left to do.

During the week, I also came across some bugs in XMPP server software, which were a great opportunity to come in contact with other developers outside of the igniterealtime project. Everybody was nice and helpful. Its really pleasant to work with the community.

Lastly, I also got in touch with some other GSoC students, which is super fun. I really enjoy meeting people from other countries and it turns out, that everybody puts their pants on the same way.

I’ll post updates roughly on a weekly basis to document the course of events. Untill then :)

Happy Hacking!

Monday, 08 May 2017

Visiting Kamailio World (Sold Out) and OSCAL'17

DanielPocock.com - fsfe | 14:50, Monday, 08 May 2017

This week I'm visiting Kamailio World (8-10 May, Berlin) and OSCAL'17 (13-14 May, Tirana).

Kamailio World

Kamailio World features a range of talks about developing and using SIP and telephony applications and offers many opportunities for SIP developers, WebRTC developers, network operators and users to interact. Wednesday, at midday, there is a Dangerous Demos session where cutting edge innovations will make their first (and potentially last) appearance.

Daniel Pocock and Daniel-Constantin Mierla at Kamailio World, Berlin, 2017

OSCAL'17, Tirana

OSCAL'17 is an event that has grown dramatically in recent years and is expecting hundreds of free software users and developers, including many international guests, to converge on Tirana, Albania this weekend.

On Saturday I'll be giving a workshop about the Debian Hams project and Software Defined Radio. On Sunday I'll give a talk about Free Real-time Communications (RTC) and the alternatives to systems like Skype, Whatsapp, Viber and Facebook.

Edit as project and Flowhub live mode

Henri Bergius | 00:00, Monday, 08 May 2017

In Flowhub you can create and edit full flow-based programming projects. The live mode enables introspecting running FBP systems. This weekend we rolled out Flowhub 0.19 which makes it easy to move between these modes.

Live mode

Flowhub’s live mode is designed for making software introspectable — flow-based systems provide access to the graph and network state information, and tools like Flowhub then visualize it.

Opening a system in live mode can be done with a URL that contains the connection details to the runtime. This means runtimes can make their live mode discoverable in many ways, from printing it at system start-up to a NFC tag or QR Code on a deployed device:

<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/EdgeSDFd9p0" width="560"></iframe>

In live mode, users can see the graph structure of the running software and the packets flowing through it. And with the right permissions, you can also navigate the possible sub-graphs and component source code.

Edit as project

To make changes to the running software, hit the Edit as project button. Flowhub will download the currently running software from the runtime and turn it into a project.

MsgFlo in live mode

In nutshell:

  • Live mode provides a read-only view to a running system
  • Project mode allows live programming a full project including its graphs and components
  • Edit as project can be used to turn a live mode session into an editable project
  • Projects can be synchronized with GitHub

Edit as project also works with the example programs that ship with Flowhub.

Sunday, 07 May 2017

Agile, inner source, open development, open source development

Inductive Bias | 15:53, Sunday, 07 May 2017

Last week I had the honour of giving a keynote at Adobe's Open Source Summit EU in Basel. Among many interesting talks they hosted a panel to answer questions around all things open source, strategy, inner source, open development. One of the questions I found intersting is how inner source/ open development and agile are related.

To get everyone on the same page, what do I mean when talking about inner source/ open development? I first encountered the concept at ApacheCon EU Amsterdam in 2008: Bertrand Delacretaz, then Day Software, now Adobe was talking about how to apply open source development principles in a corporate environment. A while ago I ran across the term http://innersourcecommons.org that essentially wants to achieve the same thing. What I find intersting about it is the idea of applying values like transparency, openess to contributions as well as concepts like asynchronous communication and decision making at a wider context.

What does that have to do with Agile? Looking at the mere mechanics of Agile like being co-located, talking face to face frequently, syncing daily in in person meetings this seems to be pretty much opposite of what open source teams do.

Starting with the values and promises of Agile though I think the two can compliment each other quite nicely:

"Individuals and interactions over processes and tools" - there's a saying in many successful popular projects that the most important ingredient to a successful open source project is to integrate people into a team working towards a shared vision. Apache calls this Community over code, ZeroMQ's C4 calls it "People before code", I'm sure there are many other incarnations of the same concept.

Much like in agile, this doesn't mean that tools are neglected alltogether. Git was created because there was nothing quite like this tool before. However at the core of it's design was the desire for ideal support for the way the Linux community works together.

"Working software over comprehensive documentation" - While people strive for good documentation I'd be so bold as to suggest that for the open source projects I'm familiar with the most pressing motivation to provide good documentation is to lower the number of user questions as well as the amount of questions people involved with the project have to answer.

For projects like Apache Lucene or Apache httpd I find the documentation that comes with the project to be exceptionally good. In addition both projects are supported with dead tree documentation even. My guess would be that working on geographically distributed teams who's members work on differing schedules is one trigger for that: Given teams that are geographically distributed over multiple time zones means that most communication will happen in writing anyway. So instead of re-typing the same answers over and over, it's less cost intensive to type them up in a clean way and post them to a place that has a stable URL and is easy to discoveri. So yes, while working software clearly is still in focus, the lack for frequent face to face communication can have a positive influence on the availability of documentation.

"Customer collaboration over contract negotiation" - this is something open source teams take to the extreme: Anybody is invited to participate. Answers to feature requests like "patches welcome" typically are honest requests for support and collaboration, usually if taken up resulting in productive work relationships.

"Responding to change over following a plan" - in my opinion this is another value that's taken to the extreme in open source projects. Without control over your participants, no ability to order people to do certain tasks and no influence over when and how much time someone else can dedicate to a certain task reaciting to changes is core to each open source project cosisting of more than one maintaining entity (be it single human being or simply one company paying for all currently active developers).

One could look further, digging deeper into how the mechanics of specific Agile frameworks like Scrum match against what's being done in at least some open source projects (clearly I cannot speak for each and ever project out there given that I only know a limited set of them), but that's a topic for a another post.

The idea of applying open source development practices within corporations has been hanging in the air for over 15 years, driven by different individuals over time. I think now that people are gathering to collect these ideas in one place and role them out in more corporations discussions around the topic are going to become interesting: Each open source community has their own specifics of working together, even though all seem to follow pretty much the same style when looked at from very high above. I'm curious to what common values and patterns can be derived from that which work across organisations.

How a slow disk affects your system

Evaggelos Balaskas - System Engineer | 10:26, Sunday, 07 May 2017

The problem

The last couple weeks, a backup server I am managing is failing to make backups!

The backup procedure (a script via cron daemon) is to rsync data from a primary server to it’s /backup directory. I was getting cron errors via email, informing me that the previous rsync script hasnt already finished when the new one was starting (by checking a lock file). This was strange as the time duration is 12hours. 12 hours werent enough to perform a ~200M data transfer over a 100Mb/s network port. That was really strange.

This is the second time in less than a year that this server is making problems. A couple months ago I had to remove a faulty disk from the software raid setup and check the system again. My notes on the matter, can be found here:

https://balaskas.gr/blog/2016/10/17/linux-raid-mdadm-md0/

Identify the problem

So let us start to identify the problem. A slow rsync can mean a lot of things, especially over ssh. Replacing network cables, viewing dmesg messages, rebooting servers or even changing the filesystem werent changing any things for the better. Time to move on the disks.

Manage and Monitor software RAID devices

On this server, I use raid5 with four hard disks:

# mdadm --verbose --detail /dev/md0


/dev/md0:
        Version : 1.2
  Creation Time : Wed Feb 26 21:00:17 2014
     Raid Level : raid5
     Array Size : 2929893888 (2794.16 GiB 3000.21 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun May  7 11:00:32 2017
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ServerTwo:0  (local to host ServerTwo)
           UUID : ef5da4df:3e53572e:c3fe1191:925b24cf
         Events : 10496

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       6       8       48        2      active sync   /dev/sdd
       5       8        0        3      active sync   /dev/sda

View hardware parameters of hard disk drive

aka test the hard disks:

# hdparm -Tt /dev/sda


/dev/sda:
 Timing cached reads:   2490 MB in  2.00 seconds = 1245.06 MB/sec
 Timing buffered disk reads: 580 MB in  3.01 seconds = 192.93 MB/sec

# hdparm -Tt /dev/sdb


/dev/sdb:
 Timing cached reads:   2520 MB in  2.00 seconds = 1259.76 MB/sec
 Timing buffered disk reads: 610 MB in  3.00 seconds = 203.07 MB/sec

# hdparm -Tt /dev/sdc

/dev/sdc:
 Timing cached reads:   2512 MB in  2.00 seconds = 1255.43 MB/sec
 Timing buffered disk reads: 570 MB in  3.01 seconds = 189.60 MB/sec

# hdparm -Tt /dev/sdd

/dev/sdd:
 Timing cached reads:     2 MB in  7.19 seconds = 285.00 kB/sec
 Timing buffered disk reads:   2 MB in  5.73 seconds = 357.18 kB/sec

Root Cause

Seems that one of the disks (/dev/sdd) in raid5 setup, is not performing as well as the others. The same hard disk had a problem a few months ago.

What I did the previous time, was to remove the disk, reformatting it in Low Level Format and add it again in the same setup. The system rebuild the raid5 and after 24hours everything was performing fine.

However the same hard disk seems that still has some issues . Now it is time for me to remove it and find a replacement disk.

Remove Faulty disk

I need to manually fail and then remove the faulty disk from the raid setup.

Failing the disk

Failing the disk manually, means that mdadm is not recognizing the disk as failed (as it did previously). I need to tell mdadm that this specific disk is a faulty one:

# mdadm --manage /dev/md0 --fail /dev/sdd
mdadm: set /dev/sdd faulty in /dev/md0

Removing the disk

now it is time to remove the faulty disk from our raid setup:

# mdadm --manage /dev/md0 --remove  /dev/sdd
mdadm: hot removed /dev/sdd from /dev/md0

Show details

# mdadm --verbose --detail /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Wed Feb 26 21:00:17 2014
     Raid Level : raid5
     Array Size : 2929893888 (2794.16 GiB 3000.21 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun May  7 11:08:44 2017
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ServerTwo:0  (local to host ServerTwo)
           UUID : ef5da4df:3e53572e:c3fe1191:925b24cf
         Events : 10499

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       4       0        0        4      removed
       5       8        0        3      active sync   /dev/sda

Mounting the Backup

Now it’s time to re-mount the backup directory and re-run the rsync script

mount /backup/

and run the rsync with verbose and progress parameters to review the status of syncing

/usr/bin/rsync -zravxP --safe-links --delete-before --partial --protect-args -e ssh 192.168.2.1:/backup/ /backup/

Everything seems ok.

A replacement order has already been placed.

Rsync times manage to hit ~ 10.27MB/s again!

rsync time for a daily (12h) diff is now again in normal rates:

real    15m18.112s
user    0m34.414s
sys     0m36.850s
Tag(s): linux, mdadm, md0

Saturday, 06 May 2017

Using Emoji for fingerprint verification

vanitasvitae's blog » englisch | 18:31, Saturday, 06 May 2017

The messaging app Telegram recently introduced end-to-end encrypted voice calls. As most of you probably know, encryption without verification is pretty useless since there is the risk of man-in-the-middle attacks. I don’t want to get too much into details about this. The point I want to make is, that you should verify your partners fingerprint (hash of the used key) in order to be secure.

The interesting part of Telegrams new feature is the way they verify fingerprints. Traditionally you are presented with a String of (typically hexadecimal – 0-9,A-F) characters. In the case of Conversations the fingerprint are 64 hexadecimal characters. Telegram on the other hand introduced the way of displaying 4 out of a set of 333 emojis (1). Note that this is only used to verify that the current voice call is secure. The next call would have a different fingerprint, so keep in mind, that we are talking about two different use cases here.

Still, how do those two methods compare? Could we use emoji in conversations to verify the fingerprint of identity keys?

With telegrams emoji encoding, there are 333⁴ = 12.296.370.321 possible hash values. This is tiny compared to the number of possibilities with the conventional fingerprint encoding (64 hex characters), which sum up to 16⁶⁴ = 115.792.089.237.316.195.423.570.985.008.687.907.853.269.984.665.640.564.039.457.584.007.913.129.639.936 unique hash values. This is far more secure than the system used by telegram. To be precise, it is 9.416.769.844.639.765.662.078.012.249.277.305.077.454.163.692.443.706.456.867.173.918.282 times more secure (theoretically).

But could we use emoji in eg. Conversations?

Lets say, we want to use emojis for fingerprint verification without trading away security. We’d have to display 31 emojis in order to be as secure as displaying 64 hex chars. Since most people are more familiar with numbers and the letters A-F, I doubt that this brings any benefits (we just cut the length of the string in half).

But what if we chose from a bigger set of emojis?

Lets say we want the fingerprint to be as short as the one in telegram (4 characters), but have the same security properties as the conventional 64 digit hex string. In order to encode the same amount of information in 4 symbols as we could in 64 hex characters, we’d have to use a pool of 18.446.744.073.709.551.616 symbols. Unfortunatelly there aren’t so many characters, let alone emojis.

But what would be the middle ground?

If we want our fingerprint to be 16 characters long, our character pool would be 65536 symbols. Thats the whole unicode space. Since there are many unicode characters that look alike and there are also a lot of “holes” in the unicode space, there are fewer usable characters.

In conclusion, it is not really possible/valuable to use emojis for fingerprint representation without trading away security.

(1): https://core.telegram.org/techfaq#q-how-are-voice-calls-authenticated

Tuesday, 02 May 2017

Installing Baobáxia at the II International Festival of Technoshamanism

agger's Free Software blog | 16:58, Tuesday, 02 May 2017

A scene from a short film created as part of the festival's video workshop.

On November 9 2016, I and my son Johannes left Denmark for Brazil in order to co-organize and attend the II International Festival of Technoshamanism. You can read more about the background for this festival at the technoshamanism site as well as in previous posts on this blog.

Each participant in the festival was expected to propose an activity, and as one of the organizers I was no exception. The II International Festival of Technoshamanism took place in the Pataxó village Pará in the extreme south of the Brazilian state of Bahia, and my proposal was to install a working node (mucúa) of the Baobáxia system in the village. The purpose of this node is to act as an archive of the indigenous cultural production, a way yo protect and salvage the indigenous culture in electronic form for decades to come, and a way for the Pataxó to do so with complete ownership over the infrastructure as well as the content – independently of corporate and proprietary content-sharing sites such as Facebook and YouTube.

Who are the Pataxó?

Well, I’ve written about them before, after I participated in the first technoshamanism festival in 2014. The Pataxó are an indigenous people (what we sometimes call “Indians”, indeed they sometimes call themselves that) who live near the sea in the extreme south of the Brazilian state of Bahia. The Aldeia Pará (Pará Village) is located in what the Pataxó call their origin or Aldeia Mãe, the area from which they were all expelled in the massacre of 1951. It is situated far from everything in the middle of an indigenous reserve comprising some 8500 hectares and is home to some 69 families. The neighboring village of Barra Velha is located 5 km from Pará and has a population of about 400 families.

The Pataxó are an intelligent and open-minded people, and though they have not had much exposure to computers, they are very conscious of the potential of modern technology in the struggle for their culture and their land, which has been a constant factor in Pataxó life for several decades – not least given that many of their villages are threatened by expulsion. In October 2016, the village of Aratikum some 100 kilometers north of Pará was actually razed to the ground by the police in the service of local land owners.

Opening ritual in Akurinã Pataxó's healing space.

The Pataxó’s motivation for hosting the event was to be able to exchange knowledge – to learn something from all us urban hackers and activists with our experience in free software and art and social movements, and to teach us something in the bargain. Each participant would propose an activity – a workshop, an art installation, a ritual, a performance, a talk – and would be free to participate in the other proposed activities. Among the activities proposed by the Pataxó were the festival’s opening and closing rituals, a healing tent, traditional Pataxó cooking, traditional body paint, fishing and hunting methods, the cultivation of manioc and manufacturing of manioc flour, musical sessions and the jogos indigenas, the indigenous sports games which took the form of a competition between Indians and non-Indians.

As I said, my proposal was to install a node of Baobáxia and – just as, or even more, important – give workshops in how to use and maintain the system (running on a dedicated Debian GNU/Linux server) themselves so it could be useful also after we left.

Technoshamanism, ancestrality and the Destructor

But if the Pataxó’s motivation for hosting the event was to make friends and allies and gain new knowledge, then what was our motivation for organizing it? And by “we” I mean a large group of people in the technoshamanism network, mainly but (obviously enough) not exclusively Brazilian: Fabi Borges, Jonatan Sola,  Sue Nhamandu, Rafael Frazão, Fernando Gregório, Luiza Só, Rodrigo Krul and too many more to mention, apart from the many participants who arrived at the festival with similar motives and gave many outstanding contributions?

One thing is the connection of technoshamanism with reconnection  - reconnecting with the Earth, reconnecting with the ancestral worldviews of the thousands of generations of people who lived close to the Earth in a mainly oral culture. The Pataxó live in a reserve where they can live well off the earth, they have a strong connection to their ancestral way of life as well as, quite literally, to their ancestors and other ancestral spirits, who often show up at their rituals.  As such, the Pataxó have 500 years of experience in dealing with European colonizers and the usurping civilization, and they have developed an immense skill in navigating this kind of pressure without losing neither their independence nor their traditional culture.

As opposed to that, the norm in our cities is that of disintegration, not least of community spirit and ancestral culture. Traditional songs and tales which might have been handed down in subtly changing ways for hundreds or thousands of years are replaced with comic books and cinema, which are replaced with endless children’s TV shows, which as we grow up are replaced with “breaking news”, X Factor and a host of even more diluted and inane TV shows. Culture ceases to be something we do ourselves and do together, neighborhoods cease to be communities and the cultural divide even splits up the families, so that we end up as disjoint individuals in a sea of strangers who can only struggle to recreate something vaguely resembling a genuine community. Sometimes, of course, as in the case of many successful free software projects, genuinely succeeding.

In his recent novel “Jerusalem“, the  writer and comic book author Alan Moore metaphorically describes this phenomenon, which he has experienced first hand in his home town of Northampton, as “the Destructor”. The Destructor was a garbage incinerator which for decades was actually and physically located in the poorest neighborhood in Northampton, reducing people’s life expectancy with at least ten years, its location a daily reminder to the inhabitants of the Boroughs of how little the rest of the city cared about them.

And yet the Boroughs was actually the oldest neighborhood in Northampton and home to a bafflingly rich, orally transmitted ancestral working class culture which was, after World War I and under the impression of the Russian revolution, deliberately crushed by city planning. According to Alan Moore, the policies which have disempowered modern Europeans by stripping us of our communities were deliberately inflicted. Moore describes the destructive effect on the communities with these words:

He saw a hundred old men and old women moved from the condemned homes where they’d raised their families, dumped in distant districts with nobody that they knew and failing to survive the transplant. By the dozen they keeled over on the well-lit stairs of their new houses; in the unfamiliar indoor toilets; onto their unprecedented fitted carpets; on the pillows of magnolia-painted bedrooms that they failed to wake to. Countless funerals fell into the Mayorhold’s fires, and furtive teenage love-affairs, and friendships between relocated children sent to different schools. Infants began to understand that they would probably now never marry the classmate they had been expecting to. All the connecting tissue, the affectations and associations, became cinders. (p. 731)

The Northampton neighborhood known as the Boroughs descended into complete misery and insecurity, containing the points of trade that “supplied the customers who drew the girls, who brought the pimps, who dealt the drugs, which bred the guns that shot the kids who lived in the house that crack built” (p. 691). But there’s a point in that – that kind of misery is very common in urbanized Brazil as well as in Europe, and a contact with people who still retain an orally transmitted culture and whose communities were never fragmented by the Destructor could teach us something about reconnecting, with the Earth and its spirits, with our natural spirituality and with true community.

The Festival area

As we arrived in Pará about November 14, our first job was to establish a good contact with the Pataxó, organize food for the event and start rigging the computers and other technical equipment.

As everybody else, we were camping in what was at first quite precarious conditions due to the heavy rainfall before the festival started.

Luckily, the Pataxó were very helpful and we managed to secure everything against the rain before the start of the festival on November 22.

Installing Baobáxia

Community Radio sending from the Pataxó Kijeme Cultural, home of the GNU/Linux computers

Before our arrival, the Pataxó had built a completely new house for cultural production, in which they had placed four stationary computers they had received from the reservation’s Fisherman’s Association which originally got them from a government program. These four computers were quite old and had Windows installed. Our first task was to replace that with GNU/Linux.

At first, our attempts at setting up the computers were haunted by technical difficulties. First of all, we were unable to get them to boot from USB drives, which meant we had to buy burnable CDs or DVDs. When we got them, we realized they could not really boot from the DVDs either due to our images being 64 bit, and these trusty old computers were actually 32 bit. We couldn’t use the Internet for troubleshooting since there was no Internet yet – it was supposed to arrive during  the week before the festival, but the roads were closed because of the rain.

In the end, Pablo Vieira from the Assentamento Terravista near Ilhéus (with the microphone in the picture above) arrived, and as it turned out, he knows these computers very well; they can boot from USB if a rather obscure BIOS setting is enabled. In his pocket was a bootable USB with the most recent 32 bit Linux Mint, and everyone was happy and the computers were well prepared for the arrival of the Internet later that week.

The Internet arriving at Aldeia Pará. Pataxó warrior Txayhuã is painting festival organizer Fabi Borges while the operator's car has stopped at the new culture house. Half an hour later, there was Internet.

I was not alone in the task of installing Baobáxia and giving workshops about it – Vincenzo Tozzi from the Mocambos network, Sicilian and founder of the Baobáxia project, joined the festival as well. Vince is a programmer and computer scientist and wrote a major part of the Baobáxia system himself, but he is really a philosopher of networks with important insights in the potential of free software and offline digital communications, and his presence was an invaluable contribution to the festival.

Vincenzo Tozzi from the Mocambos networks explains Baobáxia to village chief Ubiratã. Also listening are Pablo Vieira and Arapaty Pataxó.


Our two workshops in Baobáxia were a huge success, and especially the younger generation of the Pataxó showed a great interest in working with this technology. The Baobáxia node we installed is still active in the village and is still not connected to the Internet, but you can see the contents in its present degree of synchronization here.

What else was in the festival?

A lot of things.

Some very beautiful rituals:
DSC07216-460

And video workshops, radio workshops, capoeira, samba in the church in honor of Saint Benedito, seed exchange, agroforestry, construction of dry composting toilets, radio production, discussions about the pros and cons of ecoturism, and much, much more. I might do a followup post on that, in the meantime let it be said that the festival was a unique experience and I’m very happy to be one of the people who made it happen.

Nextcloud is about collaboration

Free Software – Frank Karlitschek_ | 10:04, Tuesday, 02 May 2017

Today the Nextcloud community released the Nextcloud 12 beta. The final release will be out later this month. This is a major new step forward. And it is also an interesting release because we are entering a new area for the product.
At the beginning, 7 years ago the focus was clearly file sync and share. Of course this term did not exist at the time or at least I didn’ know it. The task was to syncronize file between all your devices and share it with others.
This is obviously still the core of Nextcloud and Nextcloud 12 comes with all the usual iterative improvements. For example additional security hardening and improved two factor authentication, significant performance improvements especially for big installations, a ton of bugfixes and more powerful sharing features to make sharing easier and more powerful.

But it is clear that basic file sync and share is not enough anymore. People don’t want to see files popup or change on their devices without knowing what, why, when, who did this. They need ways to communicate and collaborate around the files that are shared. So the obvious next step is to add context, communication and collaboration features to the basic file sync and share.

Nextcloud already has added these areas. The focus of 12 was to combine them together into a powerful collaboration and communication platform. Basically a fully open source and self hosted Office 365 or Google G Suite alternative.

So Nextcloud comes with powerful Calendar, Contacts and Email apps. There are apps for chat, tasks management, Kanban style planning, notes taking and many other things. The goal was to bring them all together. In Nextcloud 12 you can click on any avatar and directly start a chat conversation, start a video call via WebRTC or send an email. This is all grouped around the right context. So you can do this while sharing or editing a file. With the help of Collabora Online you can also do collaborative editing of office documents together.
There is a new contacts drop down in the main menu where you directly find your colleagues and friends and can interact with them. Of course you can easily find them, even if they are on a different Nextcloud server with the help of our new Lookup server and federated APIs. It is now super easy to switch between apps with the help with the new apps navigation in the header.
The concepts of collaboration, communication and context also include the Nextcloud mobile apps for iOS and Android. In both you can now see your activity stream to see who is doing what on the server. There are also new instant push notification to the phones if important events happen like someone left a comment to a file, sent you a share invitation or invited you to a video call.

Of course this kind of integration around communication and collaboration should extend to other areas like your Desktop. This is why I’m super happy to see plugins like this KDE plugin to share directly from within your Plasma Desktop. Also the Nextcloud integration that the GNOME community is doing for calendar, contacts, maps and more is really awesome.

More information about Nextcloud 12 including some videos can be found here.

Let’s work together to create an open source alternative to the closed products from Microsoft or Google.

Everyone is invited to contribute to Nextcloud. Nextcloud is 100% open source without proprietary parts. Contributors don’t need to sign a contributor license agreement or give up any rights on their code..

The planning of the next release has already started so join us here on github to contribute with ideas, code or anything else.

Monday, 01 May 2017

Adoption of Free Software PDF readers in Italian Regional Public Administrations (fourth monitoring)

Tarin Gamberini | 08:30, Monday, 01 May 2017

The following monitoring shows that, in the last semester, nine Italian Regions have reduced advertisement about of proprietary PDF readers on their website, and that a Region has raised increased its support to for Free Software PDF readers.

Continue reading →

Saturday, 29 April 2017

Protecting your Authoritative PowerDNS Server with dnsdist

Evaggelos Balaskas - System Engineer | 23:27, Saturday, 29 April 2017

PowerDNS

My Authoritative PowerDNS configuration, is relatively simple:

Configuration

Here is my configuration:

# egrep -v '^($|#)' pdns.conf

guardian=yes
launch=bind
bind-config=/etc/pdns/named.conf
local-address=MY_IPv4_ADDRESS
local-ipv6=MY_IPv6_ADDRESS
setgid=pdns
setuid=pdns

Bind Backend

I am using a bind backend because I used to run a bind dns server and I am too lazy to change it.

the named.conf doesnt have much:

zone "balaskas.gr" IN {
    type master;
    file "/etc/pdns/var/balaskas.gr";
};

Logs

Today, I’ve noticed some unusual traffic to my server, so I’ve enabled the logging features:

log-dns-details=yes
log-dns-queries=yes
query-logging=yes

DDoS

The horror !!!

In less than 10minutes or so, almost 2500 “unique” IPs were “attacking” my auth-dns with random queries.

Let me give you an example:

utmzcnqjytkpmnop.madingyule.net
gdqlozsdqngdidkb.madingyule.net
wrojktwlwhevwtup.madingyule.net
enozexazqxoj.madingyule.net
izahejotetwlkhql.madingyule.net

IPtables

iptables to the rescue:

iptables -I INPUT -m string --algo bm --string "madingyule" -j DROP

Any dns query with the string madingyule will be blocked in INPUT chain with Boyer–Moore string search algorithm.

dnsdist

I need a more permanent solution than reading logs and block attacks with iptables, so I’ve asked the IRC about it. They pointed me to dnsdist.

I’ve already knew about dnsdist but I always thought it was a solution for recursors and not for auth-ns.

I was wrong! dnsdist is a highly DNS-, DoS- and abuse-aware loadbalancer and works fine for auth-ns setup too.

pdns configuration

My auth-ns configuration had to change to something like this:

any-to-tcp=no
disable-tcp=yes
dname-processing=yes
guardian=yes
launch = bind
bind-config = /etc/pdns/named.conf
local-address=127.0.0.1
local-port=5353

Disabling any global listener and tcp.

dnsdist configuration

here is my dnsdist configuration:

/etc/dnsdist/dnsdist.conf

-- accept DNS queries on UDP and TCP
addLocal("MY_IPv4_IP:53")
addLocal("[MY_IPv6_IP]:53")

-- fwd queries to localhost
newServer({address="127.0.0.1:5353"})

-- resets the list to this array
setACL("::/0")
addACL("0.0.0.0/0")

I am not 100% sure about the ACL but everything seems ok.

Thats it !!!! - Finished

dnsdist - client

To connect to the dnsdist daemon, you need to add the below configuration:

controlSocket("127.0.0.1")

That means, after reloading the daemon, you can connect on it with:

# dnsdist -c

Extra

Logs

 -- log everything
 addAction(AllRule(), LogAction("/var/log/dnsdist.log", false, true, false))

Domain Blocking

Let’s start with the above iptables example:

addDomainBlock("wanbo88.net.")

addDomainBlock("madingyule.net.")

You can connect to dnsdist client (see above) and and any domain you wan to block without restarting your dnsdist service.

Allow Action

Another trick you can do, is to create some custom rules by allowing any DNS queries for your domains and drop any other dns query. You can do this with something like that:

addAction(makeRule("balaskas.gr.")   , AllowAction())
addAction(makeRule("ebalaskas.gr.")  , AllowAction())

addAction(AllRule()                  , DropAction())

Rule Order

Just remember, that the rules will be processed in line order of the file.

Block ANY

You can drop all ANY queries with:

addAction(QTypeRule(dnsdist.ANY), DropAction())

although I dont recommend it.

Rate-Limiting - QPS (Queries Per Second)

Now to the good stuff: rate limiting

A simple rule is something like the below:

-- drop queries exceeding 5 qps, grouped by /24 for IPv4 and /64 for IPv6
addAction(MaxQPSIPRule(5, 24, 64), DropAction())

If you want to drop everything when they pass the 5qps:

addAction(MaxQPSIPRule(5), DropAction())

Delay

An alternative approach is to delay everything for more than 5qps (rate limiting), this may make the bot (ddos) to overlook you.

-- Delay for 1000ms aka 1s for 5qps
addDelay(MaxQPSIPRule(5), 1000)

File Descriptors

Working on a VPS (virtual private server), I’ve troubled with file descriptors.
Message in logs from dnsdist is:

Warning, this configuration can use more than 1057 file descriptors, web server and console connections not included, and the current limit is 1024

From the command line you can tweak it to 2048 like this:

# ulimit -n 2048

If you need to make it permanent:

vim /etc/security/limits.conf

*        -        nofile        2048

Traffic

okei, it’s time to see what’s the traffic:

topQueries(20,2)

will report the domains that are reaching to our dnsdsist.

topQueries() will report everything

topQueries(20,1)

will report TLD (Top Level Domains)

Identify your traffic:

grepq("balaskas.gr")

Monit

So dnsdist is now in front of my powerdns auth-ns setup and handles everything, blocking what is necessary.

To be sure that the daemon is up and running:

/etc/monit.d/dnsdist.monit

check process dnsdist with pidfile /var/run/dnsdist.pid
    alert evaggelos_AT_balaskas_DOT_gr only on { timeout, nonexist }
    start program = "/etc/init.d/dnsdist start"
    stop program  = "/etc/init.d/dnsdist stop"

dnsdist - basics

Some basic commands about dnsdist (when connecting to the client):

Commands:

addAction(                         addAnyTCRule()                     addDelay(
addDisableValidationRule(          addDNSCryptBind(                   addDomainBlock(
addDomainSpoof(                    addDynBlocks(                      addLocal(
addLuaAction(                      addNoRecurseRule(                  addPoolRule(
addQPSLimit(                       addQPSPoolRule(                    addResponseAction(
AllowAction()                      AllowResponseAction()              AllRule()
AndRule(                           benchRule(                         carbonServer(
clearDynBlocks()                   clearQueryCounters()               clearRules()
controlSocket(                     DelayAction(                       DelayResponseAction(
delta()                            DisableValidationAction()          DropAction()
DropResponseAction()               dumpStats()                        exceedNXDOMAINs(
exceedQRate(                       exceedQTypeRate(                   exceedRespByterate(
exceedServFails(                   firstAvailable                     fixupCase(
generateDNSCryptCertificate(       generateDNSCryptProviderKeys(      getPoolServers(
getQueryCounters(                  getResponseRing()                  getServer(
getServers()                       grepq(                             leastOutstanding
LogAction(                         makeKey()                          MaxQPSIPRule(
MaxQPSRule(                        mvResponseRule(                    mvRule(
newDNSName(                        newQPSLimiter(                     newRemoteLogger(
newRuleAction(                     newServer(                         newServerPolicy(
newSuffixMatchNode()               NoRecurseAction()                  PoolAction(
printDNSCryptProviderFingerprint(  QNameLabelsCountRule(              QNameWireLengthRule(
QTypeRule(                         RCodeRule(                         RegexRule(
registerDynBPFFilter(              RemoteLogAction(                   RemoteLogResponseAction(
rmResponseRule(                    rmRule(                            rmServer(
roundrobin                         setACL(                            setAPIWritable(
setDNSSECPool(                     setECSOverride(                    setECSSourcePrefixV4(
setECSSourcePrefixV6(              setKey(                            setLocal(
setMaxTCPClientThreads(            setMaxTCPQueuedConnections(        setMaxUDPOutstanding(
setQueryCount(                     setQueryCountFilter(               setRules(
setServerPolicy(                   setServerPolicyLua(                setServFailWhenNoServer(
setTCPRecvTimeout(                 setTCPSendTimeout(                 setUDPTimeout(
setVerboseHealthChecks(            show(                              showACL()
showDNSCryptBinds()                showDynBlocks()                    showResponseLatency()
showResponseRules()                showRules()                        showServerPolicy()
showServers()                      showTCPStats()                     showVersion()
shutdown()                         SpoofAction(                       TCAction()
testCrypto()                       topBandwidth(                      topClients(
topQueries(                        topResponseRule()                  topResponses(
topRule()                          topSlow(                           truncateTC(
unregisterDynBPFFilter(            webserver(                         whashed
wrandom                            addACL(                            

dnsdist - ACL

Keep in mind that the default ACL is:


> showACL()
127.0.0.0/8
10.0.0.0/8
100.64.0.0/10
169.254.0.0/16
192.168.0.0/16
172.16.0.0/12
::1/128
fc00::/7
fe80::/10
Tag(s): dnsdist, powerdns

Friday, 28 April 2017

My Kid Will Never Hack Linux

free software - Bits of Freedom | 17:43, Friday, 28 April 2017

My Kid Will Never Hack Linux

When I was a teenager, around 15-16 years old, I started to learn C programming by reading the source code for the Linux kernel. Mind you, the kernel was quite a bit smaller then and quite easy to understand. For any software over 25 years old, I think it's remarkable how legible and understandable the kernel still is.

But there's no denying the kernel is somewhat more complex today. With limited programming experience, the kernel wouldn't be your first choice of code to make sense of. The discussions are on a higher level and the threshold to contribute is much higher than it once was.

Children growing up today won't learn C programming by reading the Linux kernel source code. As technology evolve, the challenges which are at a suitable level for budding programmers move around to new and different areas, at other levels of complexity.

If you don't have any way to make two computers at point A and point B talk to each other, you'd be inclined to start with establishing a physical layer for communication: get some cables and wire them up to the computers. Once you have that, the problem is largely solved. Sure, you can always improve, but by and large you have a working physical connection between the computers.

Your children will thank you for it, and move on to figuring out how to use that physical layer to actually make some data flow back and forth between the nodes. Once you have that, you move on to the next challenge, and so forth.

There will always be some for whom fixing a broken device driver in the Linux kernel is attractive enough that they get involved with contributing to it, and learn in the process. But by and large, the challenges suitable for young programmers are elsewhere to be found.

To meet that new generation of developers, we can't rely on finding them where we once were. We need to look towards what children today find as challenges. That is where we need to be, to help them meet those challenges with open tools.

And we need to adapt our thinking accordingly: we don't need free and open source software and copyleft so that the next generation can hack the Linux kernel; we need it so they can hack a flying car.

Tuesday, 25 April 2017

FSFE Fellowship Representative, OSCAL'17 and other upcoming events

DanielPocock.com - fsfe | 12:57, Tuesday, 25 April 2017

The Free Software Foundation of Europe has just completed the process of electing a new fellowship representative to the General Assembly (GA) and I was surprised to find that out of seven very deserving candidates, members of the fellowship have selected me to represent them on the GA.

I'd like to thank all those who voted, the other candidates and Erik Albers for his efforts to administer this annual process.

Please consider becoming an FSFE fellow or donor

The FSFE runs on the support of both volunteers and financial donors, including organizations and individual members of the fellowship program. The fellowship program is not about money alone, it is an opportunity to become more aware of and involved in the debate about technology's impact on society, for better or worse. Developers, users and any other supporters of the organization's mission are welcome to join, here is the form. You don't need to be a fellow or pay any money to be an active part of the free software community and FSFE events generally don't exclude non-members, nonetheless, becoming a fellow gives you a stronger voice in processes such as this annual election.

Attending OSCAL'17, Tirana

During the election period, I promised to keep on doing the things I already do: volunteering, public speaking, mentoring, blogging and developing innovative new code. During May I hope to attend several events, including OSCAL'17 in Tirana, Albania on 13-14 May. I'll be running a workshop there on the Debian Hams blend and Software Defined Radio. Please come along and encourage other people you know in the region to consider attending.

What is your view on the Fellowship and FSFE structure?

Several candidates made comments about the Fellowship program and the way individual members and volunteers are involved in FSFE governance. This is not a new topic. Debate about this topic is very welcome and I would be particularly interested to hear any concerns or ideas for improvement that people may contribute. One of the best places to share these ideas would be through the FSFE's discussion list.

In any case, the fellowship representative can not single-handedly overhaul the organization. I hope to be a constructive part of the team and that whenever my term comes to an end, the organization and the free software community in general will be stronger and happier in some way.

Project practices compatible with OpenChain

free software - Bits of Freedom | 07:51, Tuesday, 25 April 2017

Project practices compatible with OpenChain

At the last Open Source Leadership Summit in February, I had the chance to develop some of my thoughts around OpenChain™ and its applicability for the free and open source software community at large. To recap, OpenChain is a Linux Foundation project currently headed up by Shane Coughlan (who used to work with the FSFE and is still part of our legal team). It's vision is a software supply chain for free and open source software which is trusted and with consistent license information. The OpenChain Conformance Specification1 is a "set of requirements and best practice for open source organizations to follow in an attempt to encourage an ecosystem of open source software compliance."

One way to look at this is as a checklist of things organisations must do or make available in order to have a reasonable chance of living up to its compliance commitments in free software. This includes educating software staff about free software, having a process to track free software used so the corresponding source code can be made available, and that offers to do so are included with software (where required). All in all, these are some reasonable expectations of what an organisation is expected to do. It doesn't guarantee compliance, but it encourages it.

However, as the saying goes, no chain is stronger than its weakest link. The vision of OpenChain speaks of the software supply chain, and the supply chain starts with a free and open source software project. Like the Linux kernel, or the GNU C Library. If we expect that those using free and open source software projects comply with the license terms, we must consider the project as part of the supply chain, and find ways in which we can make it as a trusted part of the chain.

When I was young, I worked summers at Ericsson Telecom AB as a tester for passive components. The way it worked is Ericsson would buy components for its products, and when the components arrived, a few of each batch would be sent for testing where the components were meticulously tested to make sure they conformed with the specification.

After my second year there, we started getting less components to test. During my third year, most of the tests had been replaced and the components went straight into storage. What replaced the tests? Trust. By working with their suppliers, Ericsson set up systems to encourage the suppliers to take a larger responsibility for the components it shipped. As the trust increased between Ericsson and its suppliers, and as suppliers took more responsibility for what they shipped, the need for Ericsson to do the testing on their side reduced drastically.

The free and open source sector right now is in a state where validation and due diligence of software licenses must be done at almost every link in the chain. OpenChain is one of the ways in which the trust can be increased, but we need to move this back to the project level and encourage reasonable compliance practices for every major free and open source software project.

We can actually look at OpenChain as providing some of the answers to what those practices would be. We need to think more broadly, but a lot of the practices mandated by OpenChain for organisations make a lot of sense for projects as well.

Have a documented FOSS policy

Every project should to have the license choice of the project documented, both as it relates to the actual source code and any ancillary parts (where the license choice may be different). The rationale for a certain license may be included, as well as information about how future license changes would be considered by the project.

This policy should also consider dependencies against external projects and their licenses.

Make FOSS training available

Developers, even in volunteer projects, need to have some basic knowledge about free software compliance in order to make informed decisions. This should be included as part of any request to sign copyright assignments and developers should be encouraged to occasionally review the project's recommended training materials about compliance.

Identify a FOSS liaison

There should be a publicly communicated e-mail address or similar contact point to which people can submit questions or concerns regarding compliance. In its simplest form, this can easily be an email address which creates ticket in a ticket system available for core developers (or another identified group). As legal information may be submitted through this, it should generally not be immediately publicly accessible though.

Arrange an external source of legal expertise

If we have a FOSS liaison function for the project which is composed of the core developers or another identified group, we need to resolve the question of escalation: if a compliance question can not be responded to, where does it escalate?

Every project should have identified an external source of legal expertise which function both as a source of expertise as needed and also as an escalation path to ensure matters reported are dealt with accurately.

Take care when distributing

To the extent the license used require it (as is usually the case when distributing binaries, which some projects may optionally do in addition to its source code), the project also need to consider the obligations placed on it when making releases.

The policy earlier described should ideally cover these cases too, but the actual work and what needs to be done can include:

  • Maintain archival copies of dependencies' source code to be able to satisfy requests for the complete corresponding source code,
  • Information about the build process,
  • Preparing and including SPDX metadata,
  • Including relevant credits and license information,
  • Including an offer for source code.

Conclusion

If you look at the current OpenChain Conformance Specification, some of the practices above are reflected in that specification. This makes sense: if free and open source software projects follow similar practices and processes as the organisations using the software, we increase the likelihood of a consistent, compliant, supply chain.

  1. The OpenChain working group is currently preparing its next release of its specification and related work.

Sunday, 23 April 2017

Augsburger Linux-Infotag: small in revenue, big in fun!

MHO | 18:41, Sunday, 23 April 2017

For the first time after a pause of several years, I allowed myself to enjoy my participation in FSFE’s booth in Augsburg. With a remarkable team of 7 (!) volunteers, we presented a rich assortment of information material and FSFE’s current merchandise. As we didn’t have any new merchandise items compared to last year, we sold only six T-shirts, one bag and one pen. So one could ask: was it worth having a booth there at all?

Of course it was! We don’t visit conferences for the money, we visit them for the people! And indeed, we had a lot of interesting discussions with visitors, it was great for us to meet the other volunteers at the booth (many of us hadn’t met before), and we enjoyed talking to the people of our booth neighbour KDE and all the other projects present. What made the event a very special pleasure was our really large team, which allowed for all volunteers to attend talks, go for lunch or just take a break whenever they wanted, always being sure that others are there to hold the fort meanwhile. Personally, I enjoyed being part of this team, meeting smart and enthusiastic people, and having a lot of fun with them!

flattr this!

Friday, 21 April 2017

We've passed the point of peak GNU

free software - Bits of Freedom | 06:59, Friday, 21 April 2017

We've passed the point of peak GNU

Some years ago by my reckoning, we passed peak GNU. It went by quicker than some of us noticed, but it was anticipated. By peak GNU, I refer to the impact of the GNU project on free and open source software, both in terms of technology and in terms of license choices.

The GNU project, more than any, have pioneered the idea of copyleft licenses through the GNU GPL. It has also been a strong advocate for quality. When people were hired to work for the GNU project in the mid 1980s, the code was scrutinized in excruciating detail for conformance to the coding standards.

When I worked as a webmaster for the GNU project in the end of the 1990s, I experienced the same attention to detail when it came to writing. In my interviews with former staff for the GNU project, one developer noted how he had never, ever, neither before nor after, in all his working career in the industry, seen the same obsession with quality as that which he saw when he joined the GNU project.

But no program lasts forever. Several of the packages which I would consider make up the core of the GNU project increasingly find themselves sidelined: Emacs doesn't have the same appeal as it once did, bash is losing its status as a default shell, gcc has increased competition and the entire tool chain looks different in many projects. What was once GNU, isn't new any more.

While the evaluation guidelines of the GNU project talk about the need for programs adopted by the GNU project to work together, and how GNU is not just a collection of useful programs, I believe the reality is different. Looking at the collection of GNU programs, there are a part of them which are clearly developer focused and serve the GNU tool chain (automake, gcc, etc).

Another part are general programs which you would assume to be part of a normal operating distribution, which the GNU project has always sought to build: dia, gnome, gpaint, and so on.

Then there are the outliers: gneuralnetwork, melting, health, and so on. All worthy programs in their own right, but neither developer tools nor something you expect as part of a standard operating system distribution.

The GNU project has a lot of potential, and a lot of good will. There's a need for projects driving copyleft adoption and showing how copyleft licensing can be a positive force for free and open source software. But there seems to be some ambivalence in how the GNU project presents itself, what programs it accepts as part of the GNU family, and how it is organised.

If nothing else, perhaps a clearer differentiation between the GNU tool chain, the GNU operating system, and additional GNU tools, would be in order.

What would be your thoughts?

Thursday, 20 April 2017

Okular 1.1 released!

TSDgeos' blog | 21:08, Thursday, 20 April 2017

Today KDE Applications 17.04 was released.

It includes Okular 1.1, it contains a nice set of features:
* Add annotation resize functionality
* Add support for auto-calculation of form contents via JavaScript
* Allow to rotate the page view using two-finger pinches on a touchscreen
* Change pages in presentation mode by swiping on touch screen
* Added support for Links that change the Optional Content visibility status
* Allow to disable automatic search while typing
* Allow to create bookmarks from the Table Of Contents

This release was brought to you by Albert Astals Cid, Oliver Sander, Luigi Toscano, Martin T. H. Sandsmark, Tobias Deiminger, Antonio Rojas, Burkhard Lück, Christoph Feck, Elvis Angelaccio, Gilbert Assaf, Heiko Becker, Hrvoje Senjan, Marco Scarpetta, Miklós Máté, Pino Toscano, Yuri Chornoivan.

Planet Fellowship (en): RSS 2.0 | Atom | FOAF |

  /127.0.0.?  /var/log/fsfe/flx » planet-en  Albrechts Blog  Alessandro at FSFE » English  Alessandro's blog  Alina Mierlus - Building the Freedom » English  Andrea Scarpino's blog  André on Free Software » English  Being Fellow #952 of FSFE » English  Bela's Internship Blog  Bernhard's Blog  Bits from the Basement  Blog of Martin Husovec  Blog » English  Blog – Think. Innovation.  Bobulate  Brian Gough's Notes  Carlo Piana :: Law is Freedom ::  Ciarán's free software notes  Colors of Noise - Entries tagged planetfsfe  Communicating freely  Computer Floss  Daniel Martí's blog  Daniel's FSFE blog  DanielPocock.com - fsfe  David Boddie - Updates (Full Articles)  Don't Panic » English Planet  ENOWITTYNAME  Elena ``of Valhalla''  English Planet – Dreierlei  English – Björn Schießle's Weblog  English – Max's weblog  English — mina86.com  Escape to freedom  Evaggelos Balaskas - System Engineer  FLOSS – Creative Destruction & Me  FSFE Fellowship Vienna » English  FSFE interviews its Fellows  Fellowship News  Fellowship News » Page not found  Florian Snows Blog » en  Frederik Gladhorn (fregl) » FSFE  Free Software & Digital Rights Noosphere  Free Software with a Female touch  Free Software –  Free Software – Frank Karlitschek_  Free Software – GLOG  Free Software – hesa's Weblog  Free as LIBRE  Free speech is better than free beer » English  Free, Easy and Others  From Out There  Graeme's notes » Page not found  Green Eggs and Ham  Handhelds, Linux and Heroes  Heiki "Repentinus" Ojasild » English  HennR's FSFE blog  Henri Bergius  Hook’s Humble Homepage  Hugo - FSFE planet  Iain R. Learmonth  Inductive Bias  Jelle Hermsen » English  Jens Lechtenbörger » English  Karsten on Free Software  Losca  MHO  Marcus's Blog  Mario Fux  Martin's notes - English  Matej's blog » FSFE  Matthias Kirschner's Web log - fsfe  Myriam's blog  Mäh?  Nice blog  Nico Rikken » fsfe  Nicolas Jean's FSFE blog » English  Norbert Tretkowski  PB's blog » en  Paul Boddie's Free Software-related blog » English  Pressreview  Rekado  Riccardo (ruphy) Iaconelli - blog  Saint's Log  Seravo  TSDgeos' blog  Tarin Gamberini  Technology – Intuitionistically Uncertain  The Girl Who Wasn't There » English  The trunk  Thib's Fellowship Blog » fsfe  Thinking out loud » English  Thomas Koch - free software  Thomas Løcke Being Incoherent  Told to blog - Entries tagged fsfe  Tonnerre Lombard  Torsten's FSFE blog » english  Viktor's notes » English  Vitaly Repin. Software engineer's blog  Weblog  Weblog  Weblog  Weblog  Weblog  Weblog  Werner's own blurbs  With/in the FSFE » English  a fellowship ahead  agger's Free Software blog  anna.morris's blog  ayers's blog  bb's blog  blog  drdanzs blog » freesoftware  egnun's blog » FreeSoftware  emergency exit  free software - Bits of Freedom  free software blog  freedom bits  gollo's blog » English  julia.e.klein's blog  marc0s on Free Software  mkesper's blog » English  nikos.roussos - opensource  pichel's blog  polina's blog  rieper|blog » en  softmetz' anglophone Free Software blog  stargrave's blog  the_unconventional's blog » English  things i made  tobias_platen's blog  tolld's blog  vanitasvitae's blog » englisch  wkossen's blog  yahuxo's blog