суббота, 16 августа 2014 г.

Semi-irregular Intresting Github Repos Ops Digest #10 (August 15)

Hi All,

New name for that digest come into my mind - Semi-irregular! It sounds good for me, let's stick with that.
I want to make some celebration with Issue #10, but first I found out that no really cool projects to show for a week - maybe because of  summer/vacation time, I don't know. So, I was waiting for one more week, and now -
Semi-irregular Intresting Github repos (for) Ops Digest, issue 10:

First, fresh docker projects:
1. docker-cheat-sheet 
Docker Cheat Sheet 
Good cheat sheet for Docker, must read.

2. flocker 
Easily manage Docker containers and their data https://clusterhq.com 
Open-source Docker orcestration service. Also, not much to say, please try tutorial.

3. flynn 
A next generation open source platform as a service (PaaS) https://flynn.io 
Ah, that's more complicated thing, but looks very promising - latyered open-sourced PaaS platform, check it out.

4. github-trending
Tracking the most popular Github repos, updated daily

Self explanatory, tracking the most popular Github repos and updating list of top objective-c, javasript and go repos in markdown output daily. You can pick other languages and run own tracker if you want.

5. cloud-ssh
Cloud enhanced SSH client replacement with host auto-completion http://leonsbox.com/cloud-ssh/

Cloud enhanced SSH client replacement with host auto-completion, espicially nice for AWS hosts.

6. cpubars
Lightweight terminal-based multicore CPU usage monitor
cpubars is a simple terminal-based tool for monitoring CPU load in real-time, especially tailored to monitoring large multicores over SSH.

7. Password-Repo
Quite good collection of password lists, which help to your services be more secure, I mean if you will deny using that obvious passwords for users.

8. mooltipass
Github repository dedicated to the mooltipass project 

Very nice project, indeed, like it very much! Aruino-based hardware password storage, powered by smartcard - wow! Fully open-sourced. Prototype is looking like that:

Ceators selling  it for 80$, but real paranoid needs to built this by own, of course.

More and more sysadmin tools starts to be written in Go language. This is collection of Go stuff for ops engineers, check it out

9. golang-devops-stuff

And useful port of psutil library for Go:
10. gopsutil
psutil for golanghttps://github.com/shirou/gopsutil

Also some vim plugins was detected by by Github radar this week:
11. vim-plug
Minimalist Vim Plugin Manager https://github.com/junegunn/vim-plug

New Vim plugin manager, like it:

12. vim-grammarous
A powerful grammar checker for Vim using LanguageTool.

"vim-grammarous is a powerful grammar checker for Vim. Simply do :GrammarousCheck to see the powerful checking"

13. vim-go
Go development plugin for Vim https://github.com/fatih/vim-go

"Full featured Go (golang) support for Vim. It comes with pre-defined sensible settings (like auto gofmt on save), has autocomplete, snippet support, improved syntax highlighting, go toolchain commands, etc... It's highly customizable and has settings for disabling/enabling features easily."
Nuff said.

Fun section:
14. carp
"interesting" VM in C. Let's see how this goes. https://github.com/tekknolagi/carp

Sample realization ov virtual machine in C. Check if you want to how to find how VM works.

15. sunfish 

Sunfish: a Python Chess Engine in 111 lines of code

"Sunfish is a simple, but strong chess engine, written in Python, mostly for teaching purposes. Without tables and its simple interface, it takes up just 111 lines of code!
The clarity of the Sunfish code provides a great platform for experimenting, be it with evaluation functions, search extensions or anything. Fork it today and see what you can do!"

16. anagramatron
twitter anagram hunter http://anagramatron.tumblr.com/

"Anagramatron hunts for pairs of tweets that use the exact same set of letters.
This script connects to the twitter stream. When it receives a new tweet it runs it through some filters, ignoring tweets that contain things like links or @mentions, or that contain less then a minimum number of characters."
You can check results of it on http://anagramatron.tumblr.com/

Small disclaimer: I'm just big fan of Github and since we have so many sources of information around I pick Github as source of information for my digest of interesting open-source projects. And because I'm Ops/Sysadmin person - I choose projects which somewhat ops-related, but not necessary. I'm not GitHub employee and this digest is not connected with GitHub in any way.

четверг, 7 августа 2014 г.

Google hiring task (first 10-digit prime in e) in bash one-liner

Hi All,
I was on a bit of short vacation, and according to my vacation rules I'm trying to not to think about my work-related stuff, or any computer-related stuff overall. I was not successfull completely with that target thanks to my smartphone :) but tried to. But during my train voyage I was quite bored, so, I was thinking about task which my friend and fellow colleague master Victor mentioned - it was famous foto of billboard which was used by Google for hiring purposes back in 2004 -

And master Victor said - "assuming you know about the e constant, solve in a Bash one-liner to get a job @Google :)" - and I was thinking about that during my trip. And I was succeded - with some remarks, of course.
Disclaimer: IMHO solving this thask in Bash is NOT GOOD, because it violates first principle of engeneering - "choosing the right tool for the job" - and bash definitely is not good for that task.
So, first I tried to find is that task could be solved in "pure bash" and I found that it is not. You can check that thorough investigation to find out that calculating e with required precision is not posible in bash because of integer overflow.
So, let's "cheat" - i.e. use some tools with bash. Which tools are suitable and not count as cheating? I think bc is absolutely required for math operations, also other unix tools, like sed and coreutils are also OK.
Let's divide task into pieces:

1. Generate e with enough precision (1000 digits)?
I found that you can easily do it with bc:
root@ubuntu:~# echo "scale=100;e(1)" | bc -l 2.718281828459045235360287471352662497757247093699959574966967627724\ 0766303535475945713821785251664274

2. We need to go through digits of e and take each 10 digits to check, then move to 1 digit forward and repeat, and so forth. We can do it using temporary file and "cut -cX..Y" syntax, but I found that using bash variable and bash string manipulation (${string:X:10}) is much convenient.

3. We need to check is current number prime or not. We can do it on "pure bash" too, but I found that we can use infamous factor tool from coreutils to do that task.
root@ubuntu:~# factor 100 100: 2 2 5 5
root@ubuntu:~# factor 177 177: 3 59
root@ubuntu:~# factor 179 179: 179

As you can see, factor tool make number factorization for any not really big integer number - it's quite enough for our purposes.
So, let's glue everything together:

And output is:
Answer is 7427466391.And we can also fold it to one-liner of course:

Misson accomplished. :)

воскресенье, 3 августа 2014 г.

Random Ops Github Digest #9 (August 3rd 2014)

Hello, my fellow readers!
I decided not ot wait until tomorrow and make new digest right now - still not sure if this digest will be weekly or more random and sporadic. :)
So, let's start, again, my favourite Github projects (sometimes Ops-like) in no particular order.

1. cheat
cheat allows you to create and view interactive cheatsheets on the command-line.
Like man command, but you'll get small cheatsheet just with commands instead big man page. Still not sure is it good thing or not....

2. awesome-django
A curated list of awesome Django apps and projects.
Another Awesome list, but for Django developers. Can be useful for Django Ops too. 

For you, fellow Puppeteers:
3. r10k 
Smarter Puppet deployment, powered by killer robots 
Relatively new tool for Puppet installations, useful if you using librarian and dynamic environments.

4. hiera-consul 
Hiera backend plugin for Consul 
If you have working Consul cluster you can get/set vualues there directly from Hiera.

Going to DevOps stuff:

5. terraform 
Terraform is a tool for building, changing, and combining infrastructure safely and efficiently. http://www.terraform.io 
"Infrastructure-as-code" tool - describe your infrastructure, deploy and control it on different providers.
It's not Puppet replacement though - http://www.terraform.io/intro/vs/chef-puppet.html - it's just more high level tool.

6. infratester 
Infrastructure Behavior Testing Framework http://infrataster.net 
Another "Infrastructure-as-code" tool - it tests infrastructure's behaviour from outside of servers:

7. devstep

Development environment builder powered by Docker and buildpacks
A dead simple, no frills development environment builder that is based around a simple goal:
I want to git clone and run a single command to hack on any software project.
 For more information please check http://fgrehm.viewdocs.io/devstep

8. libcloud-vagrant
Apache Libcloud compute provider for local Vagrant boxes
"libcloud-vagrant is a compute provider for Apache Libcloud which uses Vagrant to create VirtualBox nodes. With libcloud-vagrant installed, you could prototype a small cluster on your laptop, for instance, and then deploy it later on to Amazon, Rackspace, or any of the other clouds supported by Libcloud."

Rest of things:

9. seL4 
seL4 microkernel
Freshly opensourced L4-based microkernel, which has formal verification proof. Not really practical thing, but interesting - check its FAQ if curious.

10. snapzend 
zfs send/receive backup system http://www.znapzend.org 
Cool new backup system specifically for ZFS!

11. pghero
Database insights made easy
Personally I didn't work with more-less loaded instance of PostgreSQL for quite long time. But if you do - please check it out - it's nice web dashboard for pg:

Today's fun section is quite big:

12. cool-old-term 
A good looking terminal emulator which mimics the old cathode display 
Looks cool - but not sure is it OK for work:

13. ld-preload-soundsGenerates raw WAV output by hooking malloc() and read().

You can now hear how your compilation sounds like!

14. conway 
A real-time, persistent, multiplayer version of Conway's Game of Life 
You can roll out your own server and play with friends or play with strangers right now on http://lifecompetes.com

Small disclaimer: I'm just big fan of Github and since we have so many sources of information around I pick Github as source of information for my digest of interesting open-source projects. And because I'm Ops/Sysadmin person - I choose projects which somewhat ops-related, but not necessary. I'm not GitHub employee and this digest is not connected with GitHub in any way. 

вторник, 29 июля 2014 г.

Weekly Ops Github Digest #8 (July 29th 2014)

So, it seems that our digest becoming quite regular, it's issue number 8 and number 7 was out exactly one week ago, same as issue 6 was two weeks ago, nice.
Let's try to keep it that way - new issue every Monday/Tuesday, for fresh start of new week!
... but please note that possible upcoming vacation can interfere with my intentions. :)

OK, let's go.

1. elvish
An experimental Unix shell


Nice new experimental shell (written in Go), more traditional than Xiki but also with intresting new features, like nice autocompletion and navigation mode, built in daemon for syncing command history from different instances and new Clojure/Lisp-like built-in language. Worth a try, but only if you have Linux, it doesn't compile on Mac, so, if you have one and want to be a contributor - welcome.

2. perf-tools Performance analysis tools based on Linux perf_events (aka perf) and ftrace.
Set of performance analyzing tools for Linux, using (relatively) new Ftrace interface, written by famous Brendan D. Gregg, author of  Systems Performance book, DTrace book etc.
Will not talk much about that, in short - amazing stuff, check my previous post about it.

3. Next 3 repos connected with Docker, lets merge them together:

Fast, isolated development environments using Docker http://www.fig.sh

Describe your environment in single file and run it with single command "fig up".
Looks cool? Then check out

Simple web focused Dockerfile based PaaS server. 


Similar idea, but realization is little bit different - and idea is more general and limited at the same time. Just add Dockerfile to your webapp and after that run "git push" to deploy your project on choosen environment (Vagrant/Amazon/Digital Ocean/Rackspace/own server) - but only limited (but quite poular) languages and platforms are supported.
Next thing -

RegistratorService registry bridge for Docker

will help you with automatic "register/deregisters services for containers based on published ports and metadata from the container environment". I.e. you can automatically add/remove services to LB, for example. External service registers/pairing/discovery are supported also. Nice thing if your building scalable application on Docker.

Going to more general developer's stuff.
4. Two Git help pages. More general 

Git Cheat Sheet
and more specific


First one is normal Git cheat sheet, but second one is more like FAQ and list of common tasks and workarounds.

5. CockroachDB.
A Scalable, Geo-Replicated, Transactional Datastore

So, it looks like NoSQL is not very popular buzzword now, NewSQL is much better!
But if talking seriously it really looks promising - it took better parts from both worlds:

ALPHA for now, we'll see how it'll going.

6. pycrastinateTODO less, DO more. Keep your code clean without changing the way you code.

Pycrastinate is a language-agnostic tool that helps you keep your codebase (whether it is legacy or new) under control in a transparent way (i.e. without interfering with your coding). I.e. it checks for FIXMEs and TODOs and gently tring to force you to do something with that. :)
Check this talk or slides if interested. 

7. Streisand.
"Streisand sets up a new server running L2TP/IPsec, OpenSSH, OpenVPN, Shadowsocks, Stunnel, and a Tor bridge. It also generates custom configuration instructions for all of these services. At the end of the run you are given an HTML file with instructions that can be shared with friends, family members, and fellow activists."
Not like very funny thing, especially for persons who struggle with internet censorshop somewhere but 
worth to checking then.

And time for fun!
The Permanent Web 

According to website:
"IPFS is a global, versioned,  peer-to-peer filesystem. It combines good ideas from Git, BitTorrent, Kademlia, SFS, and the Web. It is like a single bittorrent swarm, exchanging git objects. IPFS provides an interface as simple as the HTTP web, but with permanence built in. You can also mount the world at /ipfs."
Very experimental thing for now, but maybe it's the source of new Internet, who knows?

воскресенье, 27 июля 2014 г.

Iosnoop is a great tool, and Gregg has more!

Hello !
So, did you already missing for Github digest? OK, OK, it's in progress, gimme more time please.
But now I want to talk about new exciting tool, again.
If you consider yourself as experienced Linux sysadmin and still do not reading Brendan Gregg's blog - you are making big mistake! Please follow it through RSS or follow Gregg's twitter @brendangregg right now - I will wait. :)
Before Gregg was working in Joyent and most of his posts was connected with DTrace, but now he moved to Netflix, and most of his work now connected with Linux.
So, Gregg was ported own iosnoop tool for Linux now. What is iosnoop? It's like a tcpdump, but for disk activity. It's hard to believe, but despite of its practical top of 10000 IOPS - really iosnoop is a bash script (more precisely - combination of bash with optimized awk) and it works out of the box on most modern kernels (even debug symbols not required)!
You can say - 'Ok, it's cool new small nifty tool, which provides some Kbs of text output, meh'. Not really impressive, right? But Gregg also provide cool thing to visualize that (or similar) data - heatmaps - and also provide tooling for that. Check that great post for making latency heatmap from iosnoop output.
So, let's try that stuff on something practical. I'm suspecting that newly installed KVM server is not behaving OK, I/O seems too slow. How we can detect that? Ok, let's reboot some VM, run iosnoop for 180 sec and compare heatmaps of reboot process.
Let's do that first on some normal server first:

I cap output at 100ms, just for simplicity, but it's not critical - real value took about 150 ms, because of queueing. Server was rebooted near 80 sec timepoint, and process was finished near 120 sec.
But compare it with target server:

It looks definitely not OK. Results are much more scattered, and queueing effect looks really terrible - top delay reach almost 6000 ms! Why? It's another question, I'll try to find that tomorrow. :)
One more time - please check Gregg's blog and his github repo - you can find much more cool things there!

понедельник, 21 июля 2014 г.

Ops Github Digest #7 (July 21st 2014)

Hello Everybody!
Summer is going on, it's too hot for thinking much about work and stuff, everybody is want to go to vacation, right?
Right, but for everyone else trapped in stifling offices - next issue of Ops Github Digest!

1. Anode - Utility for analyzing graphite metrics. Experimental package.https://github.com/mattrco/anode.exp Anode is a tool for experimenting with different analysis algorithms on metrics and other time series.
You have your data from Graphite, but don't know what to do with it? Try to experiment with that tool to find out: 

WARNING! DANGER AHEAD! Some math required. Please check that link for explanation how it works.

2.  More awesomeness!
We have another list of awesome lists - 
https://github.com/sindresorhus/awesome, awesome algorithms - https://github.com/tayllan/awesome-algorithms, awesome machine learning - https://github.com/josephmisiti/awesome-machine-learning, and awesome bigdata - https://github.com/onurakpolat/awesome-bigdata
But my favorite - list of awesome T-Shirt sites - https://github.com/elder-cb/teesites !

3. Log file navigator http://lnav.org
Nice log reading / colorizing tool, especially for own server.

4. Another PuppetDB frontend:
Spotify's Puppetexplorer https://github.com/spotify/puppetexplorer
Has own query language and able to combine data from different PuppetDBs into one.

5. Facette
Time series data visualization and graphing software http://facette.io/
Facette is a web application to display time series data from various sources — such as collectd or Graphite — on graphs, designed to be easy to setup and to use.

6. storm is a command line tool to manage your ssh connections. https://github.com/emre/storm/

7. maestro-ng - Orchestration of Docker-based, multi-host environmentshttps://github.com/signalfuse/maestro-ng
Quite self-explanatory, please check if you use Docker.

8. miniLock - miniLock is a small, portable file encryption software.https://github.com/kaepora/miniLock
New generation of modern file encryption software, which tries to proof that JS-crypto suck less nowadays.

9. And now - for you, my fellow esoteric programming language freaks!

Escher: A language for programming in metaphors https://github.com/gocircuit/escher Truly mind blowing if you have enough patience to wrap your mind around it. :)

понедельник, 14 июля 2014 г.

Sporadic Ops Github Digest #6 (July 14th 2014)

Hi All! 
I'm (again, still fighting with lazyness - and overcoming it) presenting our next digest:

1. cv - Linux tool to show progress for cp, rm, dd, ...
One image is better than thousand words:

2.  powerline-shell - A beautiful and useful prompt for your shell
Yes, it is nice shell from picture above:

3. awesome-perl - A curated list of awesome Perl frameworks and libraries.

At last! Perl!!!!11111

4. awesone-awesome-awesome - A a curated list of curated lists of awesome lists.
Ha-ha-ha-ha! Made my day!

And of course, there's awesome-awesome-awesome-awesome list too - for now, containing only list above...

5. More stuff for Python learning:https://github.com/rasbt/python_reference - Useful functions, tutorials, and other Python-related thingshttps://github.com/s16h/py-must-watch - Must-watch videos about Python 

6.  The automated Pokémon speedrunning machine 
An automated computer program that speedruns Pokémon, which you can run on own computer. I'm not big fan of Pokemon and speedruns, but fan of old consoles. :) Also nice piece of good Lua code with AI.