I Love creating with Water Colors!

I painted this back in 2016...

I hope you like it! 😃

 

Nike's on a Persian Rug.

Create a Cert to Sign PDFs via LibreDraw

For those using Linux as their main OS...

Some Prerequisites...

If you have not allready instlled them these packages...

    > apt install -y openssl libnss3-tools

 

First Create a Self Signed Cert...

> cd ~/

> mkdir -p pki/nssdb

> cd pki

# Create a self-signed cert

> openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out ss-pdf-x509.pem

# Initialize the nssdb database

> certutil -N -d nssdb/

# Create the p12 file...

> openssl pkcs12 -inkey key.pem -in ss-pdf-x509.pem -export -out ss-pdf-pkcs12.p12

# Place cert into nssdb

> pk12util -d nssdb/ -i ss-pdf-pkcs12.p12

# List certs in our nssdb

> certutil -L -d nssdb

 

Then, Setup LibreDraw to Find your Cert

1. Start up LibreDraw.
2. Goto: Tools -> Options -> LibreOffice -> Security -> Certificate Path -> Select NSS path
3. Choose the ~/pki/nssdb

 

Hope this helps someone!

Upgrading Hadoop 2.0 to 2.4.2 (Avoiding a broken Ambari UI)

Hortonworks Hadoop Upgrade

HDP 2.0 to 2.4.2

---

This blog is a bit of info regarding a Hadoop cluster upgrade from HDP 2.0 to 2.4.2.

Given the current state of Hadoop at the company, we are mainly focused on an upgrade of HDFS, as current production systems mainly make use of HDFS, Yarn, Mapreduce, Hive and Pig. Currently, we also run a stand-alone Spark 1.4.0 cluster that is setup in an HA mode, along with Tachyon to improve disk access among other optimizations. Spark and Tachyon are built to use HDFS by default. We run 20 Spark Worker nodes directly on Data Nodes of the cluster. We don't run Yarn NodeManagers on nodes that run Spark Workers, so that the Spark Worker Nodes are mainly dedicated to Spark. One Tachyon worker also runs on each of the Spark Worker nodes.

We have been working out the details of the Hadoop upgrade on a small 5 node Dev cluster, were we start with HDP 2.0 and test the upgrade steps to HDP 2.4.2.

The initial plan was to start with with Ambari 1.5 and HDP 2.0.
We will upgrade Ambari to 1.7 and then to 2.0. Then upgrade HDP 10 2.2.
Then Upgrade Ambari to 2.2.
Once we are at Ambari 2.2, we would upgrade HDP to 2.0.
Then upgrade HDP to 2.4.2 via Ambari.

The problem with this approach is that experience has shown that upgrades to Ambari fail 9 out of 10 times. In most cases, after incremental updates to Ambari, Ambari will get into a state where certain pulldowns, dialogs, and/or tabbed pages are missing, making it impossible to perform some typical Ambari procedure, such as adding or monitoring a given service. As with our typical experience, the incremental upgrade of Ambari failed after the upgrade from 1.7 to 2.0.
Firstly, we could not get webhcat up and running via Ambari, and Secondly, the Admin "Manage Ambari" pulldown was missing.

So I have requested to we try the upgrade, by short-cutting through the incremental upgrade steps, moving directly from 2.0 to 2.4.2. Luckily, HortonWorks' latest upgrade to the HDP core (2.4.2) includes an upgrade path from 2.0 to 2.4.2, at least from a manual upgrade angle. This is was very good news to me when I heard and read about it. So here is what the "shortcut" plan entails...

1. On all of the local drives of the name-nodes, journals, and data-nodes, we move the hadoop data dirs to a read-only backup dir. All of or data in hadoop is located on JBOD drives. We have 10 drives per node and we mount these drive to /grid/01 to 10. Within each /grid/[n]/ dir, hadoop creates its data/meta-data directory "hadoop". We will move these hadoop/ dirs to hadoop_2.0.bk/ and make the dirs read-only.
2. We make note of all config data that contains paths to /grid/[n]/hadoop.
3. We make note of the location of all services by node.
4. Next, we backing-up all configs in the current PRD Ambari-DB (Postgres) and in /etc//conf, and the current Hive metadata (MySql).
5. We then remove all prior Hadoop and HDP packages from each node.
6. We install the latest version of Ambari (2.4.2)
7. We install all new HDPO 2.4.2 services to the cluster, ensuring that services are placed on the same nodes as the old cluster, and as mapped in step 3.
8. We bring up Hadoop 2.4.2 and test it.
9. We enable HA following the same service map as the old cluster.
10. We shutdown all of HDFS.
11. We run a script that renames all of the /grid/[n]/hadoop/ dirs to /grid/[n]/hadoop_2.4.2.init.
12. We run a script that renames all of the /grid/[n]/hadoop_2.0.bk/ dirs /grid/[n]/hadoop/.
13. We start up the journal nodes.
14. We start up the NN1 manually with the -upgrade option. NN1 will first upgrade the Journal Nodes and the Name Node data structures. Then NN1 will tell the data nodes to perform data structure upgrades.
15. We start up all of the Data Nodes (which will receive a command from the NameNode to perform an upgrade). Thus, Data Nodes will upgrade in parallel.
16. We start up the StandBy NameNode (NN2) with the options "-bootstrapStandby -force"
17. The name node should become active once all data nodes have upgraded.
18. If the NN nodes not become active, if it make sense (based on the name-node logs), to force the NN out of standby mode... We run: sudo -u hdfs hdfs dfsadmin -safemode forceExit
19. Once the active NN is out of standby mode, run smoke-tests.

This upgrade process has been tested on our Dev Hadoop cluster. This is a good solution for our Production cluster as we are mainly concerned with an upgrading HDP to the Latest. Using this process, our original Hadoop configurations are lost. But we don't mind, as we understand how to optimize HDFS and MapReduce, and it will be easy given the latest version of the Ambari UI. Our Spark cluster will be replaced with Spark on Yarn, given the latest version of HDP, which also brings up the the latest version of Spark 1.6, and allows us to make use of our cluster resources more effectively.

I also have to add here, the steps that we used to upgrade our Hive MetaStore to the latest version.
So, initially, we attempted to use the following steps (to a failed result)...

1. Shutdown Hive and Hive MetaStore.
2. Restore the old Hive MetaStore SQL DB (containing our original metadata).
3. Execute Hive MetaStore upgrade scripts to upgrade MySql hive schema from 0.12 to 1.2.1.2.4.

Every thing looked good until we tried to perform a group-by select, on a partition field. A group-by on a partition field would result in a error during the map-reduce process, where map-reduce would complain that existing partitions were ambiguous. So it seemed that some strange change to the Hive MetaStore table was causing subtle and bad effects. In fact, the deletion of a partition resulted in all of /apps/hive/warehouse/ getting deleted. This was really bad... After wrestling and failing with attempts at reviewing and patching MetaStore Schema in the MySql database, we decided to take a different approach... Something that starts us off at a clean pristine state, with regard to the Hive MetaStore...

1. Start with the latest pristine version of the Hive MetaStore schema for 1.2.1.2.4.
2. Capture the "create-table output" from the original HDP 2.0 production system, the "show create table " Hive command.
3. Run this create table command command on the new system (HDP 2.4.2), to recreate our tables.
4. Capture the "show partitions " output from the old cluster (HDP 2.0).
5. Generate "Alter Table Add Partition" commands based on that output.
6. Run the generated "Alter Table Add Partition" on the new cluster (HDP 2.4.2) to re-generate the tables partitions.
7. Now, this worked with no issues. But you have to understand what you are doing when you Hive tables are "managed-tables". With "managed-tables", your tables can not be created if the directory already exists within /apps/hive/warehouse, so we also have a step where we rename /apps/hive/warehouse to /apps/hive/warehouse.save (and make is read-only), and we create an empty /apps/hive/warehouse/ dir, before running all of the create-table script, what will restore all of the Hive MetaStore meta-data for our tables, as they were when we have (HDP 2.0) installed. After all of the tables are created, we do a directory rename on the /apps/hive/warehouse dirs to flip our our data dir (that contains the actual data), back into position, and we restore write access to it again.
8. Finally, we run a set of hive queries to ensure that all of our data can be accessed via Hive again, in the HDO 2.4.2 install.

These steps are merely an outline of what needs to be done in general, there will be more details to you particular run-book, based on your cluster environment, and based on what you discover during upgrade dry-runs that you will be doing on a development cluster... You will be doing dry-runs... Won't you.

The bottom line is that, a clean install of a particular version of Ambari and it's corresponding database, followed by a clean install of the HDP version of choice, is the best path to a clean running Ambari and HDP install, given the current state of the install/upgrade process.
My experience has been that the most important aspect of the upgrade process, preserving and upgrading HDFS "just works", (will it better work, or you can forget about it...). So we leverage that aspect of the upgrade process to by pass all of the headaches associated to incremental upgrades to Ambari.

We will exercise, this upgrade process on our dev cluster a few more times, them we will do it to production cluster at the end of the week, outside of production hours.
So stay tuned... I will update this post after the production system upgrade.

I'm hoping that this data helps you to have a safe, pleasant upgrade experience.

Hadoop upgrades can be tricky, so exercise your theories and create a run-book of what works on a test cluster. Then exercise your run-book a few times, on your dev cluster, to get all of the mysteries and kinks out. 

2016-06-20
-Sidlo

Successful Production HDP 2.0 to 2.4.2 Upgrade Completed

We started the upgrade process on Friday 23-Jun, and had a completely successful upgrade with no data loss as of 25-Jun. The upgrade steps and process took a total of 28hrs. Besides my team, two awesome ThinkBig consultants, Serhiy Blazhiyevskyy and Cedric Barnett where a huge help in ensuring the success of the upgrade.

One thing that we noticed was that installs of clients were taking a long time, and then later, restarts of services were taking a long time. After some debugging, we found that there was a cluster level variable called fetch_nonlocal_groups, that is by default, set to True. The problem with this default is that on client installs and on services startups, Hadoop's group names are checked for non-locally, or over ldap. So, if you have ldap based authorization set up on you cluster, the and, if you don't have those Hadoop groups setup in your ldap servers, ldap will perform an exhaustive search which can timeout, and cause the startup of install process to fail. Before we discovered that the fetch_nonlocal_groups variable can be set to False on your cluster's blue print via Ambari's configs.sh script, we had to suffer through a slow and tedious install which we had to retry about 4 times before it was finally successful. Then we had to suffer through slow service startups, before we finally figured out how to correct the situation. If you have a similar experience or environment, use the configs.sh script to change the fetch_nonlocal_groups variable for you cluster blueprint to False. During a cluster install, it may be possible to perform the variable update before services are pushed to nodes by Ambari, but I am not an expert in Ambari's internals, so I am just guessing.

One useful tip, with regard to something unexpected that can come up during this type of production upgrade, is to find that hards drive have failed, or are failing enough for some services to fail. We had 8 drives fail on 8 nodes, and on one other node, we had 8 out of 10 drives fail. In my experience, a DataNode will not start up if even 1 drive has failed on the node (even if node failure tolerance has been set to 2 or 3). Now, to ensure that we have no dataloss, we needed to start up with all good drives attached to running DataNodes and upgrade with all good drives that are available. So, we need to have DataNodes that have bad drives start up, by having them dis-regard any drives that are bad. Luckily, this can be done through Ambari's configuration groups. Being able to use Configuration Groups to get DataNodes to start up even when they have bad drives was a real life...data-saver.

2016-06-25

-Sidlo

Decommissioning DataNodes on HDP 2.0

Issues & Adventures with Decommissioning Data Nodes on HDP 2.0

Use the Ambari UI to decommission...

1. Decommission the HBase RegionServer
2. Decommission the NodeManager
3. Finally Decommission the DataNode

The HBase RegionServer and NodeManager statuses will to to "Decommissioning" and eventually to "Decommissioned".

Use have to use the Ambari Rest API to see that the DataNode is on its way to being Decommissioned...

> curl -s --user admin:40rt0n http://prdslsldsafht25.myfamilysouth.com:8080/api/v1/clusters/prdslsldsafht/hosts/prdslsldsafht12.myfamilysouth.com/host_components/DATANODE | less

Look for "desired_admin_state" : "DECOMMISSIONED".

What seems to be the correct steps...
One would think that Decommissioning should simply work, but it seems that there are bugs that require the procedure to be specific...

1. First make the Active NameNode NN1
2. Perform the decomm of the DN (Via the Ambari UI).
1. You will notice that NN2 does not recognize the decomm of the DN.
2. There is a bug that indicates that the Decomm action might be taken by a random NN,
   Which may invalidate my hypothesis...
1.   https://issues.apache.org/jira/browse/AMBARI-4927

What goes wrong if Decomm occurs with NN2 and the Active NN...

• If the Active is NN2 and Standby is NN1...
• The decommissioning of a node, is only recognized by NN1, NN2 will continue to write to the datanode as NN1 trys to ensure that all blocks are replicated off of it. NN2 is at fault as no more new blocks should be sent to a decommissioning node.
• Thus it is possible for Standby-NN1 to be decommissioning a DN while Active-NN2 is writing to the DN.
• It may be that HA decommissioning tests are always running with Active-NN1 and Standby-NN2 and never the other way around.

Decommissioning nodes would not fully decommission because a very few blocks continue to be indicated as under-replicated, but I could not find those blocks and files via fsck to handle them.

NN UI...
| Decommissioning Datanodes : 2
| Node Transferring
| Address Last
| Contact Under Replicated Blocks Blocks With No
| Live Replicas Under Replicated Blocks
| In Files Under Construction Time Since Decommissioning Started
| prdslsldsafht12 10.211.25.122:50010 2 232143 0 4 0 hrs 8 mins
| prdslsldsafht13 10.211.25.123:50010 2 261441 0 3 0 hrs 8 mins

hdfs fsck...
|  Total dirs:    140574
|  Total files:   5367953
|  Total symlinks:                0 (Files currently being written: 5353)
|  Total blocks (validated):      5683249 (avg. block size 40621873 B) (Total open file blocks (not validated): 1796)
|  Minimally replicated blocks:   5683249 (100.00001 %)
|  Over-replicated blocks:        385172 (6.7773204 %)
|  Under-replicated blocks:       0 (0.0 %)
|  Mis-replicated blocks:         0 (0.0 %)
|  Default replication factor:    3
|  Average block replication:     3.0742598
|  Corrupt blocks:                0
|  Missing replicas:              0 (0.0 %)
|  Number of data-nodes:          75
|  Number of racks:               5
| FSCK ended at Thu Nov 13 21:26:31 MST 2014 in 1680353 milliseconds


I simply pull the nodes, making them dead-nodes by turning off the DN process.
Stangely, Ambari shows that the nodes are in a decommissioned state.
But, I still have the option to stop the DataNode service, so I do that.
The NN UI continues to indicate that the 2 nodes are still in the process of decommissioning.
So, we need to make the NN think that the node is dead by removing the hostnames from dfs.exclude and running -refreshNodes.
After running -refreshNodes, the nodes no-longer display in the Decommissioning nodes list.
And, the main NN UI page shows that 2 nodes are Decommissioned.

Thus, we can by-pass the Decommissioning process if it can't resolve the last few under replicated blocks.

The following link was helpful in that it indicated that it was possible for the decommission process to completely stall on a final few under-replicated blocks. One should be able to find the files associated those under replicated blocks and perform an appropriate handling on then. But, in our case, fsck was not finding any under-replicated blocks at all.
http://stackoverflow.com/questions/17789196/hadoop-node-taking-a-long-time-to-decommission

Extend a laptops Keyboard and Mouse to other Computers/Displays

I've recently set myself up with a trading system that consists of 2 large displays and 2 laptops and their displays.
I am using both laptops. The first laptop is used mainly to display trading charts, and to develop and upgrade my indicator to NinjaTrader 7. I don't do real trades with NinjaTrader 7, as it is still Beta software.
The second laptop, is used to display trading charts, and to do real trading, using the stable version of NinjaTrader 6.5.
The problem is that it can be very confusing to move between 2 mice and 2 laptop keyboards. So , I Googled for a way of extending a computers laptop and keyboard over to another computer. Essentially, VNC without the display.

The solution that I found was a project called Synergy-Plus. This code allows you to set up your main interface-computer (the one that you sit in front of most of the time), as the server, and another computer as a client. Once, the software is set up, when your mouse hits the left of right edge of you monitor, the mouse and keyboard control jumps over to the other computer and allows you to control its mouse, keyboard and windows.
Synergy-Plus is has really reduced the confusion in the area by allowing me to control both laptops through a single laptop's mouse and keyboard.

Synergy-Plus supports various Operating Systems including Windows, Linux and MAC.
I am currently using it with windows 7 64-bit with no problems so far.

Solaris: Cron Jobs Don't Run (!bad user) locked account

It took a bit of time to find out that my cron jobs where not running for a given user on a Solaris box.

Looking at /var/cron/log I was seeing "!bad user" (userName).

But, the user was fine, I was using it, I'd already su'ed to become the user several times.

Turns out that the problem is that, cron does not like users that have been locked out due to not changing the password. I never saw a request to change the password because I always su to become the user rather than logging in as that user.

So, to fix the problem, run "passwd -u userName" as root or via sudo. After that, the cronjobs run fine.

Apparently, the fact that on Solaris, cron does not run the jobs of a locked user, is not documented in any visible manner.

The Acer Aspire Revo R3610 as a Linux Server.

For many years I have been building Linux servers for my own web and mail services.
I've always limited by hardware costs to $500 or less by reusing old equipment.
Over the last several months, I have had hard drive failures on 3 of my servers, two of which I could live without. Last week, when the third system (my mail server) started to experience failures, I started looking for a replacement server.

I started to look at the PogoPlug as a possible solution. My new PogoPlug had just recently arrived in the mail, and it seemed like a good candidate as a mail server replacement. With is ultra small footprint and its low power consumption, I started hacking it with grand designs in mind. Unfortunately, after getting to the point of being able to install OpenPogo packages to a USB drive, I became cautious about making too many changes to the PogoPlugs sofware. Partly, due to the fact that I have come to enjoy what PogoPlug does best, making data on a hard drive that you have at home, available on the web for yourself and others (if you choose), in a safe a secure manner. Thus, I dropped any further major tweeking for now, until I can see a cleaner safer way of adding additional Linux services to the PogoPlug via OpenPogo, without compromising it's security or bricking it.

I took a trip to Fry's hoping to find a solution via a Shuttle X2700 mini-pc server. I only wanted to spend about $400 at most on the new server, but found that I would be at about $600 after buying the bare bones system, then adding the CPU memory and hard drive. So I lost interest in that route and looked at what they offered in terms of complete systems. Here, I came across the Acer Revo. It's an affordable mini-pc system that uses the ATOM processor, the R3610 is a 64bit processor with 2 cores. Linux actually reports 4 cores because each core can run 2 threads. I bought the R3610 for $329 plus tax. This gets me all I need for a server plus more.

The Acer Aspire Revo R3610 came with Windows 7. But I want Linux to be the main OS.
When I initially tried to boot the Ubuntu installer from a USB drive. I was disappointed to see the message "No Operating system found". I reformatted to USB drive to NTFS and used some procedures that I found on the net to make the USB drive bootable via an original Windows Vista bootable Install CD, but that did not work either as I got messages like "Bootmngr was not found." or "OS was not found.". The simple solution was just a few steps.

1. Download the iso for Ubuntu Server or Ubuntu Desktop.
   
2. Cleanly format a USB drive as FAT32.
3. Use the "Universal Netboot Installer" to place a bootable install of 64 bit Ubuntu Linux on the USB Drive. Just point the unetbootin utility to the Ubuntu iso and the USB drive letter of the newly formatted FAT32 USB drive.
4. Plug the USB drive into the Acer Revo.
5. Set the Acer Revo to boot from the USB drive.
6. Partition about half of the 160Gig drive to run Linux, and leave the other half for Windows 7.
7. Install Ubuntu...
   

Ultimately, I'd like to be able to run Windows 7 VM os to Linux via Xen.

But, I'll have to leave that experiment for later...

Acer Aspire Revo Specs:

AR3610-U9012Genuine Windows^® 7 Home Premium , Intel® Atom™ Processor N330 (1MB L2 cache, 1.60GHz, 533MHz FSB), 2GB (1/1) DDR2 800 SDRAM, 160GB SATA hard drive, multi-in-one card reader, NVIDIA® ION™ graphics, gigabit LAN, 802.11b/g/Draft-N WLAN