Open source *

I will continue the story "How to put the whole world into a regular laptop: PostgreSQL and OpenStreetMap" with secrets about OpenStreetMap geodata, on which many companies have built their business, but not everyone shares the details... Well, today we will open crucial details.

The OSM database in PosgreSQL after loading from the dump takes up more than 587 GB. This is already a large database by the standards of a DBMS, and one huge table for each type of object will not work. For manageability, such data must be partitioned, it's good that PostgreSQL supports declarative data partitioning. It remains only to figure out how to split geographical data. After searching and comparing, the H3 hierarchical hexagonal geospatial indexing system came to rescue. All this was implemented in my openstreetmap_h3 project for fast processing and loading of the world dump into the PostGIS database.

I considered following options from geopartitioning systems...

When a person used to say that he controls the whole world, he was usually placed in the next room with Napoleon Bonaparte. I hope that these times are in the past and everyone can analyze the geodata of the entire Earth and get answers to their global questions in minutes and seconds. I published Openstreetmap_h3 - my project, which allows you to perform geoanalytics on data from OpenStreetMap in PostGIS or in any query engine that can work with Apache Arrow / Parquet.

First of all, I say hello to the haters and skeptics. What I developed is really unique and solves the problem of transforming and analyzing geodata using the usual and familiar tools available to every analyst and data science specialist without bigdata, GPGPU, FPGA. What looks easy to use and code now is my personal project where I have invested my vacations, weekends, sleepless nights and a lot of personal time over the past 3 years. Maybe I will share the background of the project and the rake that I went through, but first I will still describe the end result.

Today I will discover America to you based on OpenStreetMap data in PostgreSQL15/PostGIS and my project openstreetmap_h3. Let's run the query and compare its execution time on the Citus column store in PostgreSQL and on the standard 100GB database partitioned by H3 geoindex.

We will find the top15 buildable locations in North America and the total length of roads, as well as their type and surface. I will not overload the publication with program logs, let's focus on the data! You can easily repeat all requests yourself on your laptop/computer.

Do you need to produce satellite interferometry results for your work or study? Or should you find the way to process terabytes of radar data on your common laptop? Maybe you aren't confident about the installation and usage of the required software. Fortunately, there is the next generation of satellite interferometry products available for you. Beginners can build the results easily and advanced users might work on huge datasets. Open Source software PyGMTSAR is available on GitHub for developers and on DockerHub for advanced users and on Google Colab for everyone. This is the cloud-ready product, and it works the same as do you run it locally on your old laptop as on powerful cloud servers.

Nfun is an embedded language and expression executor that supports primitive types, arrays, structures and lambda expressions.

Most likely, you have already met tasks that require such a tool, and in this article I want to show examples of its application, its capabilities and why it may be useful to you.

Overview

Once every true-linux engineer gets a trouble: there is no any software in his distro or it's built without needed options. I am keen on the phrase: "Only source control gives you freedom".

Of course, you can build this software on your computer without any src-packages, directly (with simplification: configure, make, make install). But it's a non-reproducible solution, also hard for distribution.

The better way is to make distro-aligned package that can be built if needed and that produces lightly distributed binary-packages. It's about debian-source packages(debian,ubuntu,etc), pkgbuild (for arch), ebuild for gentoo, src-rpm for red hat-based, and many others.

I will use cri-o like a specimen.

Before reading the text below I strongly recommend to get familiarized with the official Debian policy manual placed here and debhelper manpage.

Also you will be required to setup some variables like DEBMAIL and DEBFULLNAME for proper data in changelog and other places.

Global corporations became a part of our everyday life for a long time ago, their products often don’t leave an alternative option for users. Either is it exist? This article touches an issue of dominating big companies in certain areas, but also contains a row of useful tips. Spoiler of one of them: if you’re a user of Android then the tips will help you increase time between charging your phone and improve your privacy.

Overview

Sometimes all of us need to make a graphical installer for one's own linux distro. It goes without saying that you are able to use a distro-specific installer like Anaconda for RedHat-based or DebianInstaller for debian-based. On the other hand Calamares is a graphical installer which is not aligned with only one package manager.

I want to share my experience how to make a universal install solution with GUI. I did not find any complete article about it, hence, I reinvented the wheel.

We are often asked whether we send bug reports to developers of open-source projects. The answer is yes. More than that— we sometimes track the progress. This article is about one of the cases when this habit prevented a sloppy bug fix.

Ссылка на русскую версию / link to Russian version

FPGA InsideOut is an attempt to make a set of educational FPGA videos presented in the “human-in-the-loop” style. In these videos we will not only show how we are interfacing with an actual FPGA board but will also provide synchronous real-time visualisation of FPGA's internal logic.

For our first video we have picked a CRC circuit (cycle redundancy check) which is based on a linear feedback shift register. This circuit goes through several transformations during the course of the video. Intrigued? - let’s watch the video.

In 2021 we published several articles and showed you errors found in open-source projects. The year 2021 ends, which means it's time to present you the traditional top 10 of the most interesting bugs. Enjoy!

The .NET 6 turned out to be much-awaited and major release. If you write for .NET, you could hardly miss such an event. We also couldn't pass by the new version of this platform. We decided to check what interesting things we can find in the sources of .NET libraries.

Flipper Zero is an open-source multi-tool for geeks and penetration testers. It so happened that the Flipper Zero project and the PVS-Studio analyzer crossed paths. A philosophical question: should we check the project, if the project developers have already started fixing errors? Let's try to do this.

It's freezing outside, everyone has already decorated the Christmas tree and bought tangerines. New Year is coming! So, it's time to meet the Top 10 interesting bugs found by the PVS-Studio C++ analyzer in 2021.

In any history textbooks, the modern time has already been called the time of the next change of the industrial structure or the fourth industrial revolution (Industry 4.0). The main role, in this case, is given to information and IT systems. In an attempt to reduce the cost of IT infrastructure, unify and accelerate the process of developing IT solutions, humanity first invented "clouds" in order to replace traditional data centers, and then containers to replace virtual machines.

Clearly, containers appear more vulnerable from a security point of view. What are the advantages of containerization over virtualization? In fact, there are quite a lot of them:

• the possibility of more flexible use of available resources (no need to backup them as in the case of virtual machines);

• the ability to save resources (no need to spend them on many copies of the OS for each virtual machine);

• no delays at startup (just start of the process is almost instantaneous compared to the time needed to load the virtual machine);

• the interaction between processes, even if isolated, is much easier to implement when needed than between virtual machines. That is how, by the way, came the concept of microservices, which has recently become very popular.

All of the above led to the very rapid development of container technologies, despite the recurring problems with the security of already deployed container cloud systems, their hacks, and data leaks. Accordingly, the work on strengthening container security is also continuing. This is what will be discussed further in this article.

We often check large projects because it's easier to find bugs there. What if we try PVS-Studio on a small project? In this article we analyze Blend2D — a library for vector 2D graphics. Let's look at what we found.

Introduction

It's no secret that large projects have fascinating errors. It's not just "the larger the codebase is – the more errors we can find". It's also a known fact that the density of errors grows along with the codebase. That's why we love checking large projects — to treat you with a variety of "yummy" and tricky errors and typos. Besides, it's always interesting to search through a huge project with lots of dependencies, legacy code, and other stuff.

Today I'm moving away from this tradition. I decided to take a small project and see what PVS-Studio can find there. I chose Blend2D — branch master, commit c484790.

We dust off the irregular series of articles about the Chromium project check. Let's look at the code quality in the latest Chromium release and check the new features of the PVS-Studio analyzer.

Today, we discuss C# code quality and a variety of errors by the example of CMS DotNetNuke. We're going to dig into its source code. You're going to need a cup of coffee...

It's cool when enthusiastic developers create a working clone of a famous game. It's even cooler when people are ready to continue the development of such projects! In this article, we check TheXTech with PVS-Studio. TheXTech is an open implementation of the game from the Super Mario universe.

How to make a parallel book for language learning. Part 1. Python and Colab version

This is a second article on making parallel books. Today we will use the more advanced tool which will bring rich UI functionality. Lingtrain Alignment Studio is a web application written on Vue and Python. The main purpose of it is to extract the parallel corpora from two raw texts and make a bilingual (or even multilingual) parallel book. This is an open-source project and I will be glad to hear all of your bright ideas. Links to the sources and our community contacts can be found below. Los geht's!

Setup

The app is packed into the docker container. It's a simple technology to deploy your stuff anywhere from the server to your local machine. It's available across all the operating systems. So at first, you need a docker installed locally. Then you need to run two simple commands. The first will download the container:

docker pull lingtrain/aligner:v4

And the second one will run the application:

docker run -v C:\app\data:/app/data -v C:\app\img:/app/static/img -p 80:80 lingtrain/aligner:v4

C:\app\data and C:\app\img — your local folders.

The app will be available on the 80th port. Let's open the localhost page in your favorite browser.

We will make three simple steps: Load, Align, Create