Pull to refresh
103.88
Rating

High performance *

Methods for obtaining systems high performance

Show first
Rating limit

The journey of scaling up a production Elasticsearch cluster

High performance *NoSQL *Database Administration *Amazon Web Services *Distributed systems *

In this article, I will tell you about a-few-years journey of scaling the Elasticsearch cluster in production environment, which is one of the vital elements of the iPrice technology stack. 
I will describe challenges we encountered and how we approached them.

Read more
Total votes 1: ↑1 and ↓0 +1
Views 388
Comments 0

Kernel Queue: The Complete Guide On The Most Essential Technology For High-Performance I/O

High performance *System Programming *C *Development for Linux *
Tutorial

When talking about high-performance software we probably think of server software (such as nginx) which processes millions requests from thousands clients in parallel. Surely, what makes server software work so fast is high-end CPU running with huge amount of memory and a very fast network link. But even then, the software must utilize these hardware resources at maximum efficiency level, otherwise it will end up wasting the most of the valuable CPU power for unnecessary kernel-user context switching or while waiting for slow I/O operations to complete.

Thankfully, the Operating Systems have a solution to this problem, and it's called kernel event queue. Server software and OS kernel use this mechanism together to achieve minimum latency and maximum scalability (when serving a very large number of clients in parallel). In this article we are going to talk about FreeBSD, macOS and kqueue, Linux and epoll, Windows and I/O Completion Ports. They all have their similarities and differences which we're going to discuss here. The goal of this article is for you to understand the whole mechanism behind kernel queues and to understand how to work with each API.

Read more
Total votes 7: ↑7 and ↓0 +7
Views 2.3K
Comments 1

In-Memory Showdown: Redis vs. Tarantool

VK corporate blog High performance *Database Administration *Tarantool *
image

In this article, I am going to look at Redis versus Tarantool. At a first glance, they are quite alike — in-memory, NoSQL, key value. But we are going to look deeper. My goal is to find meaningful similarities and differences, I am not going to claim that one is better than the other.

There are three main parts to my story:

  • We’ll find out what is an in-memory database, or IMDB. When and how are they better than disk solutions?
  • Then, we’ll consider their architecture. What about their efficiency, reliability, and scaling?
  • Then, we’ll delve into technical details. Data types, iterators, indexes, transactions, programming languages, replication, and connectors.

Feel free to scroll down to the most interesting part or even the summary comparison table at the very bottom and the article.
Read more →
Total votes 18: ↑17 and ↓1 +16
Views 2.2K
Comments 3

The way we made an external PCIe RAM disk based on the DDR memory

High performance *Cloud computing *Data storage *Computer hardware Data storaging

RAM disk, this is a disk based on RAM memory chips. This kind of disk is not able to retain data after the power is turned off (unless a supporting battery is used), but has an exceptionally high read/write speed (especially for random access) and an unlimited lifespan. It is important in tasks that need a lot of cycles to write over information, even professional SSD drives don’t live long. To the operating system the RAM disk is indistinguishable from an SSD or HDD disk and no special drivers or setup is required. Unlike a disk that is virtually located in the computer’s RAM memory, where the maximum memory capacity is limited to 128-256 GB in the best consumer motherboards, a RAM disk for a PCIe slot, in general, has no volume limits and can work in any MB with a PCIe slot.

Read more
Total votes 3: ↑3 and ↓0 +3
Views 3.8K
Comments 0

Overview of Morris's counters

Qrator Labs corporate blog High performance *Algorithms *Mathematics *

On implementing streaming algorithms, counting of events often occurs, where an event means something like a packet arrival or a connection establishment. Since the number of events is large, the available memory can become a bottleneck: an ordinary n-bit counter allows to take into account no more than 2^n - 1events.
One way to handle a larger range of values using the same amount of memory would be approximate counting. This article provides an overview of the well-known Morris algorithm and some generalizations of it.

Another way to reduce the number of bits required for counting mass events is to use decay. We discuss such an approach here [3], and we are going to publish another blog post on this particular topic shortly.

In the beginning of this article, we analyse one straightforward probabilistic calculation algorithm and highlight its shortcomings (Section 2). Then (Section 3), we describe the algorithm proposed by Robert Morris in 1978 and indicate its most essential properties and advantages. For most non-trivial formulas and statements, the text contains our proofs, the demanding reader can find them in the inserts. In the following three sections, we outline valuable extensions of the classic algorithm: you can learn what Morris's counters and exponential decay have in common, how to improve the accuracy by sacrificing the maximum value, and how to handle weighted events efficiently.

Read more
Total votes 12: ↑12 and ↓0 +12
Views 597
Comments 0

How to build a high-performance application on Tarantool from scratch

VK corporate blog High performance *Algorithms *Lua *Tarantool *
Tutorial
image

I came to Mail.ru Group in 2013, and I required a queue for one task. First of all, I decided to check what the company had already got. They told me they had this Tarantool product, and I checked how it worked and decided that adding a queue broker to it could work perfectly well.

I contacted Kostja Osipov, the senior expert in Tarantool, and the next day he gave me a 250-string script that was capable of managing almost everything I needed. Since that moment, I have been in love with Tarantool. It turned out that a small amount of code written with a quite simple script language was capable of ensuring some totally new performance for this DBMS.

Today, I’m going to tell you how to instantiate your own queue in Tarantool 2.2.
Read more →
Total votes 18: ↑18 and ↓0 +18
Views 1.3K
Comments 0

Making a Tarantool-Based Investment Business Core for Alfa-Bank

VK corporate blog High performance *System Analysis and Design *Database Administration *Tarantool *

A still from «Our Secret Universe: The Hidden Life of the Cell»

Investment business is one of the most complex domains in the banking world. It's about not just credits, loans, and deposits — there are also securities, currencies, commodities, derivatives, and all kinds of complex stuff like structured products.

Recently, people have become increasingly aware of their finances. More and more get involved in securities trading. Individual investment accounts have emerged not so long ago. They allow you to trade in securities and get tax credits or avoid taxes at the same time. All clients coming to us want to manage their portfolios and see their reporting on-line. Most frequently, these are multi-product portfolios, which means that people are clients of different business areas.

Moreover, the demands of regulators, both Russian and international, also grow.

To meet the current needs and lay a foundation for future upgrades, we've developed our Tarantool-based investment business core.
Read more →
Total votes 14: ↑14 and ↓0 +14
Views 1.2K
Comments 0

Deploying Tarantool Cartridge applications with zero effort (Part 2)

VK corporate blog High performance *Database Administration *Distributed systems *Tarantool *
Tutorial


We have recently talked about how to deploy a Tarantool Cartridge application. However, an application's life doesn't end with deployment, so today we will update our application and figure out how to manage topology, sharding, and authorization, and change the role configuration.

Feeling interested? Please continue reading under the cut.
Read more →
Total votes 15: ↑15 and ↓0 +15
Views 738
Comments 0

Fault Tolerance Web Architecture for Our Cloud Solutions

VK corporate blog High performance *Virtualization *Cloud computing *Cloud services
image

Hi Habr,

I'm Artyom Karamyshev, a system administration team leader at Mail.Ru Cloud Solutions (MCS). We launched many products in 2019. We've aimed to make API services easily scalable, fault-tolerant, and ready to accommodate rapid growth. Our platform is running on OpenStack, and in this article, I describe all the component fault tolerance issues that we've resolved.

The overall fault tolerance of the platform is consists of its components fault tolerance. So, I'm going to show you step by step tutorial about all levels where we've found the risks.
Read more →
Total votes 25: ↑24 and ↓1 +23
Views 2K
Comments 0

Accelerating PHP connectors for Tarantool using Async, Swoole, and Parallel

VK corporate blog High performance *PHP *SQL *NoSQL *


In the PHP ecosystem, there are currently two connectors for the Tarantool server: the official PECL extension tarantool/tarantool-php written in C, and tarantool-php/client written in PHP. I am the author of the latter one.

In this article I would like to share the results of performance testing of both these libraries and show how you can achieve 3x-5x performance improvement (on synthetic tests!) with minimal changes in code.
Read more →
Total votes 39: ↑39 and ↓0 +39
Views 1.5K
Comments 0

Deploying Tarantool Cartridge applications with zero effort (Part 1)

VK corporate blog High performance *IT Infrastructure *Database Administration *Distributed systems *


We have already presented Tarantool Cartridge that allows you to develop and pack distributed applications. Now let's learn how to deploy and control these applications. No panic, it's all under control! We have brought together all the best practices of working with Tarantool Cartridge and wrote an Ansible role, which will deploy the package to servers, start and join instances into replica sets, configure authorization, bootstrap vshard, enable automatic failover and patch cluster configuration.

Interesting, huh? Dive in, check details under the cut.
Read more →
Total votes 29: ↑29 and ↓0 +29
Views 1.1K
Comments 0

Tarantool Kubernetes Operator

VK corporate blog High performance *Database Administration *Kubernetes *Tarantool *


Kubernetes has already become a de-facto standard for running stateless applications, mainly because it can reduce time-to-market for new features. Launching stateful applications, such as databases or stateful microservices, is still a complex task, but companies have to meet the competition and maintain a high delivery rate. So they create a demand for such solutions.

We want to introduce our solution for launching stateful Tarantool Cartridge clusters: Tarantool Kubernetes Operator, more under the cut.
Read more →
Total votes 34: ↑34 and ↓0 +34
Views 1.4K
Comments 0

Tarantool Data Grid: Architecture and Features

VK corporate blog High performance *Lua *Data storage *Tarantool *


In 2017, we won the competition for the development of the transaction core for Alfa-Bank's investment business and started working at once. (Vladimir Drynkin, Development Team Lead for Alfa-Bank's Investment Business Transaction Core, spoke about the investment business core at HighLoad++ 2018.) This system was supposed to aggregate transaction data in different formats from various sources, unify the data, save it, and provide access to it.

In the process of development, the system evolved and extended its functions. At some point, we realized that we created something much more than just application software designed for a well-defined scope of tasks: we created a system for building distributed applications with persistent storage. Our experience served as a basis for the new product, Tarantool Data Grid (TDG).

I want to talk about TDG architecture and the solutions that we worked out during the development. I will introduce the basic functions and show how our product could become the basis for building turnkey solutions.
Read more →
Total votes 39: ↑38 and ↓1 +37
Views 1.6K
Comments 0

Тarantool Cartridge: Sharding Lua Backend in Three Lines

VK corporate blog High performance *Lua *Distributed systems *Tarantool *

In Mail.ru Group, we have Tarantool, a Lua-based application server and a database united. It's fast and classy, but the resources of a single server are always limited. Vertical scaling is also not the panacea. That is why Tarantool has some tools for horizontal scaling, or the vshard module [1]. It allows you to spread data across multiple servers, but you'll have to tinker with it for a while to configure it and bolt on the business logic.

Good news: we got our share of bumps (for example, [2], [3]) and created another framework, which significantly simplifies the solution to this problem.

Тarantool Cartridge is the new framework for developing complex distributed systems. It allows you to concentrate on writing business logic instead of solving infrastructure problems. Under the cut, I will tell you how this framework works and how it could help in writing distributed services.
Read more →
Total votes 26: ↑25 and ↓1 +24
Views 1.7K
Comments 0

Enumerable: How to yield a business value

High performance *.NET *C# *ООP *Microservices *
This article is a brief explanation about how using a common language keywords might have an influence on the budget of IT-infrastructure of a project or help to achieve some limitations/restrictions of hosting infrastructure and, moreover, will be a good sing of the quality and maturity of the source code.
Read more →
Total votes 12: ↑11 and ↓1 +10
Views 1.7K
Comments 0

12.3 million of concurrent WebSockets

High performance *C++ **nix *Lua *
Recovery mode

One thing about WebSockets is that you need a lot of resources on the client's side to generate high enough load for the server to actually eat up all the CPU resources.


There are several challenges you have to overcome because the WebSockets protocol is more CPU demanding on the client's side than on the server's side. At the same time you need a lot of RAM to store information about open connections if you have millions of them.


I've been lucky enough to get a couple of new servers for a limited period of time at my disposal for the hardware "burnout" tests. So I decided to use my Lua Application Server — LAppS to do both jobs: test the hardware and perform the LAppS high load tests.


Read more →
Total votes 6: ↑3 and ↓3 0
Views 9.5K
Comments 0

How to speed up LZ4 decompression in ClickHouse?

Яндекс corporate blog High performance *Open source *C++ *Big Data *
When you run queries in ClickHouse, you might notice that the profiler often shows the LZ_decompress_fast function near the top. What is going on? This question had us wondering how to choose the best compression algorithm.

ClickHouse stores data in compressed form. When running queries, ClickHouse tries to do as little as possible, in order to conserve CPU resources. In many cases, all the potentially time-consuming computations are already well optimized, plus the user wrote a well thought-out query. Then all that's left to do is to perform decompression.



So why does LZ4 decompression becomes a bottleneck? LZ4 seems like an extremely light algorithm: the data decompression rate is usually from 1 to 3 GB/s per processor core, depending on the data. This is much faster than the typical disk subsystem. Moreover, we use all available CPU cores, and decompression scales linearly across all physical cores.
Read more →
Total votes 23: ↑21 and ↓2 +19
Views 9.8K
Comments 1
1

Authors' contribution