Mail.ru Group

Fault Tolerance Web Architecture for Our Cloud Solutions

• Mail.ru Group corporate blog,
• High performance,
• Virtualization,
• Cloud computing,
• Cloud services

Hi Habr,

I'm Artyom Karamyshev, a system administration team leader at Mail.Ru Cloud Solutions (MCS). We launched many products in 2019. We've aimed to make API services easily scalable, fault-tolerant, and ready to accommodate rapid growth. Our platform is running on OpenStack, and in this article, I describe all the component fault tolerance issues that we've resolved.

The overall fault tolerance of the platform is consists of its components fault tolerance. So, I'm going to show you step by step tutorial about all levels where we've found the risks.

Read more →

pushtaev February 10, 2020 at 05:47 PM

Tips and tricks from my Telegram-channel @pythonetc, January 2020

• Mail.ru Group corporate blog,
• Python,
• Programming

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications.

The order of except blocks matter: if exceptions can be caught by more than one block, the higher block applies. The following code doesn’t work as intended:

Read more →

pushtaev January 15, 2020 at 01:32 PM

Tips and tricks from my Telegram-channel @pythonetc, December 2019

• Mail.ru Group corporate blog,
• Python,
• Programming

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications.

Different asyncio tasks obviously have different stacks. You can view at all of them at any moment using asyncio.all_tasks() to get all currently running tasks and task.get_stack() to get a stack for each task.

Read more →

gen December 18, 2019 at 12:48 PM

Accelerating PHP connectors for Tarantool using Async, Swoole, and Parallel

• Mail.ru Group corporate blog,
• High performance,
• PHP,
• SQL,
• NoSQL

In the PHP ecosystem, there are currently two connectors for the Tarantool server: the official PECL extension tarantool/tarantool-php written in C, and tarantool-php/client written in PHP. I am the author of the latter one.

In this article I would like to share the results of performance testing of both these libraries and show how you can achieve 3x-5x performance improvement (on synthetic tests!) with minimal changes in code.

Read more →

pushtaev December 17, 2019 at 12:57 PM

Tips and tricks from my Telegram-channel @pythonetc, November 2019

• Mail.ru Group corporate blog,
• Python,
• Programming

Tips and tricks from my Telegram-channel @pythonetc, November 2019

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications.

PATH is an environment variable that stores paths where executables are looked for. When you ask your shell to run ls, the shell looks for the ls executable file across all paths that are presented in PATH.

Read more →

dokshina December 16, 2019 at 12:41 PM

Deploying Tarantool Cartridge applications with zero effort (Part 1)

• Mail.ru Group corporate blog,
• High performance,
• IT Infrastructure,
• Database Administration,
• Distributed systems

We have already presented Tarantool Cartridge that allows you to develop and pack distributed applications. Now let's learn how to deploy and control these applications. No panic, it's all under control! We have brought together all the best practices of working with Tarantool Cartridge and wrote an Ansible role, which will deploy the package to servers, start and join instances into replica sets, configure authorization, bootstrap vshard, enable automatic failover and patch cluster configuration.

Interesting, huh? Dive in, check details under the cut.

Read more →

sannikovdmitry December 2, 2019 at 07:31 PM

СodeSide. The new game for Russian AI Cup

• Mail.ru Group corporate blog,
• Abnormal programming,
• Sport programming,
• Entertaining tasks,
• Artificial Intelligence

pushtaev November 13, 2019 at 04:31 PM

Tips and tricks from my Telegram-channel @pythonetc, October 2019

• Mail.ru Group corporate blog,
• Python,
• Programming

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications

If you want to iterate over several iterables at once, you can use the zip function (it has nothing to do with ZIP file format):

Read more →

Tarantool Kubernetes Operator

• Mail.ru Group corporate blog,
• High performance,
• IT Infrastructure,
• Database Administration,
• Kubernetes

Kubernetes has already become a de-facto standard for running stateless applications, mainly because it can reduce time-to-market for new features. Launching stateful applications, such as databases or stateful microservices, is still a complex task, but companies have to meet the competition and maintain a high delivery rate. So they create a demand for such solutions.

We want to introduce our solution for launching stateful Tarantool Cartridge clusters: Tarantool Kubernetes Operator, more under the cut.

Read more →

Tarantool Data Grid: Architecture and Features

• Mail.ru Group corporate blog,
• High performance,
• Lua,
• Data storage,
• Distributed systems

In 2017, we won the competition for the development of the transaction core for Alfa-Bank's investment business and started working at once. (Vladimir Drynkin, Development Team Lead for Alfa-Bank's Investment Business Transaction Core, spoke about the investment business core at HighLoad++ 2018.) This system was supposed to aggregate transaction data in different formats from various sources, unify the data, save it, and provide access to it.

In the process of development, the system evolved and extended its functions. At some point, we realized that we created something much more than just application software designed for a well-defined scope of tasks: we created a system for building distributed applications with persistent storage. Our experience served as a basis for the new product, Tarantool Data Grid (TDG).

I want to talk about TDG architecture and the solutions that we worked out during the development. I will introduce the basic functions and show how our product could become the basis for building turnkey solutions.

Read more →

Rosik October 9, 2019 at 05:32 PM

Тarantool Cartridge: Sharding Lua Backend in Three Lines

• Mail.ru Group corporate blog,
• High performance,
• Server optimization,
• Lua,
• Distributed systems

In Mail.ru Group, we have Tarantool, a Lua-based application server and a database united. It's fast and classy, but the resources of a single server are always limited. Vertical scaling is also not the panacea. That is why Tarantool has some tools for horizontal scaling, or the vshard module [1]. It allows you to spread data across multiple servers, but you'll have to tinker with it for a while to configure it and bolt on the business logic.

Good news: we got our share of bumps (for example, [2], [3]) and created another framework, which significantly simplifies the solution to this problem.

Тarantool Cartridge is the new framework for developing complex distributed systems. It allows you to concentrate on writing business logic instead of solving infrastructure problems. Under the cut, I will tell you how this framework works and how it could help in writing distributed services.

Read more →

pushtaev October 8, 2019 at 11:10 AM

Tips and tricks from my Telegram-channel @pythonetc, September 2019

• Mail.ru Group corporate blog,
• Python,
• Programming

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications

Read more →

andrewbo29 September 18, 2019 at 05:22 PM

How we made landmark recognition in Cloud Mail.ru, and why

• Mail.ru Group corporate blog,
• Algorithms,
• Image processing,
• Machine learning,
• Artificial Intelligence

With the advent of mobile phones with high-quality cameras, we started making more and more pictures and videos of bright and memorable moments in our lives. Many of us have photo archives that extend back over decades and comprise thousands of pictures which makes them increasingly difficult to navigate through. Just remember how long it took to find a picture of interest just a few years ago.

One of Mail.ru Cloud’s objectives is to provide the handiest means for accessing and searching your own photo and video archives. For this purpose, we at Mail.ru Computer Vision Team have created and implemented systems for smart image processing: search by object, by scene, by face, etc. Another spectacular technology is landmark recognition. Today, I am going to tell you how we made this a reality using Deep Learning.

Read more →

pushtaev September 5, 2019 at 04:18 PM

Tips and tricks from my Telegram-channel @pythonetc, August 2019

• Mail.ru Group corporate blog,
• Python,
• Programming

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications

If an instance of a class doesn’t have an attribute with the given name, it tries to access the class attribute with the same name.

>>> class A:
...     x = 2
...
>>> A.x
2
>>> A().x
2

Read more →

blondnika August 21, 2019 at 01:46 PM

How to Make Emails and Not Mess Up: Practical Tips

• Mail.ru Group corporate blog,
• Spamming and anti-spamming,
• IT systems testing,
• Email layout,
• Web services testing

• Tutorial

A developer, who first encountered generating emails, has almost no chance to write an application, that will do it correctly. Around 40% of emails, generated by corporate applications, are violating some form of standard, and due to this, there are problems with delivery and display. There are reasons for this: emails are technically more difficult than the web, and operating emails is regulated by a few hundred standards, as well as an uncountable number of generally accepted (and not as much) practices, whereas the email clients are more varied and unpredictable than browsers. Testing may significantly improve the situation, but materials that are dedicated to testing the email system, are practically non-existent.

Mail.ru regularly interacts with its users by email. In our projects, all the components responsible for generating emails and even individual mailings, are subject to mandatory testing. In this article, we will share our experience (learning from our mistakes).

Read more →

pushtaev August 7, 2019 at 03:34 PM

Tips and tricks from my Telegram-channel @pythonetc, July 2019

• Mail.ru Group corporate blog,
• Python,
• Programming

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications

You can’t mutate closure variables by simply assigning them. Python treats assignment as a definition inside a function body and doesn’t make closure at all.

Read more →

sismetanin August 1, 2019 at 01:35 PM

Contextual Emotion Detection in Textual Conversations Using Neural Networks

• Mail.ru Group corporate blog,
• Python,
• Data Mining,
• Big Data,
• Machine learning

Nowadays, talking to conversational agents is becoming a daily routine, and it is crucial for dialogue systems to generate responses as human-like as possible. As one of the main aspects, primary attention should be given to providing emotionally aware responses to users. In this article, we are going to describe the recurrent neural network architecture for emotion detection in textual conversations, that participated in SemEval-2019 Task 3 “EmoContext”, that is, an annual workshop on semantic evaluation. The task objective is to classify emotion (i.e. happy, sad, angry, and others) in a 3-turn conversational data set.

Read more →

kitashov July 12, 2019 at 11:11 AM

AI-Based Photo Restoration

• Mail.ru Group corporate blog,
• Algorithms,
• Image processing,
• Machine learning

Hi everybody! I’m a research engineer at the Mail.ru Group computer vision team. In this article, I’m going to tell a story of how we’ve created AI-based photo restoration project for old military photos. What is «photo restoration»? It consists of three steps:

• we find all the image defects: fractures, scuffs, holes;
  
• we inpaint the discovered defects, based on the pixel values around them;
  
• we colorize the image.
  

Further, I’ll describe every step of photo restoration and tell you how we got our data, what nets we trained, what we accomplished, and what mistakes we made.

Read more →

lenkis July 10, 2019 at 01:29 PM

Lua in Moscow 2019: Interview with Roberto Ierusalimschy

• Mail.ru Group corporate blog,
• Programming,
• Lua,
• Interview

Some time ago, the creator of Lua programming language, Roberto Ierusalimschy, visited our Moscow office. We asked him some questions that we prepared with the participation of Habr.com users also. And finally, we’d like to share full-text version of this interview.

Read more →

pushtaev July 4, 2019 at 01:35 PM

Tips and tricks from my Telegram-channel @pythonetc, June 2019

• Mail.ru Group corporate blog,
• Python,
• Programming

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications

The \ symbol in regular string have special meaning. \t is tab character, \r is carriage return and so on.

You can use raw-strings to disable this behaviour. r'\t' is just backslash and t.

You obviously can’t use ' inside r'...'. However, it still can be escaped by \, but \ is preserved in the string:

Read more →