tag:blogger.com,1999:blog-23744466095652228272024-03-06T11:02:26.322+05:30Technical StuffThis blog is all about technical questions in C, C++, data structures like linked lists, Binary trees, UNIX and other software developement issues which I faced.Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.comBlogger186125tag:blogger.com,1999:blog-2374446609565222827.post-27501057679406581432020-08-12T20:50:00.003+05:302020-08-13T08:00:08.159+05:30My docker learnings<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" trbidi="on">Recently I started working on a project which includes both NodeJS and Python application. We planned to Deodorization build process for our project. During Deodorization I faced multiple issues which I eventually solved. I thought of writing a post on my docker experience to help others. Below are the some of my observations/issues I found during my journey</div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b>1. Using multiple FROM in docker file:</b> Never ever use two FROM instructions in the Dockerfile. Docker will take only last FROM instruction. Use one FROM instruction and install another framework. In my case I use FROM for NodeJS and installed python as shown below.</div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><span style="color: red;">WRONG WAY:</span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihSpHLM2hmSE8rNduHHfm24GVOr6RgF_o6CW9SbKgDVkkeAzeRARcU4qWh3nNt8irnymv3x5lveDVClBynoA9QbIj46b9VkEZ-N2cGw-z_7iLgmg-2PAlHCYZYurnS8D2itasJhZ-1POXk/s1234/wrong_multiple_from_docker.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="970" data-original-width="1234" height="258" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihSpHLM2hmSE8rNduHHfm24GVOr6RgF_o6CW9SbKgDVkkeAzeRARcU4qWh3nNt8irnymv3x5lveDVClBynoA9QbIj46b9VkEZ-N2cGw-z_7iLgmg-2PAlHCYZYurnS8D2itasJhZ-1POXk/w328-h258/wrong_multiple_from_docker.png" width="328" /></a></div><div class="separator" style="clear: both; text-align: left;"><span style="caret-color: rgb(255, 0, 0);"><span style="color: #2b00fe;">CORRECT WAY:</span></span></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSqQ42P3S7NdBChAzBKye_zgeHL8uLMeGtH-ZYYgVCjFM7NaB0zXFxKEv6EEFs-1RBjXI9x4uNglIc8t0kMXEMjDuPMStOnN9s_9d_zqA4bTcBDKbLsedjHPX9VttmzXiyi-YljEfXzC7K/s1292/correct_from_docker.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1176" data-original-width="1292" height="466" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSqQ42P3S7NdBChAzBKye_zgeHL8uLMeGtH-ZYYgVCjFM7NaB0zXFxKEv6EEFs-1RBjXI9x4uNglIc8t0kMXEMjDuPMStOnN9s_9d_zqA4bTcBDKbLsedjHPX9VttmzXiyi-YljEfXzC7K/w513-h466/correct_from_docker.png" width="513" /></a></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b>2. Never copy all files at the same time:</b> This is really funny, When you install dependencies after copying all the files, dependencies will install every time when you run docker build. </div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><span style="color: red;">INCORRECT WAY:<div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjS28ZZd0k2to92uY6LB134TeEhk6Nf8wparbxqMTNZHlqMs_sXVDKF6IS5xymuO6lPsK6yQkMm-ybskF5YFOB7oXnvIvgHqPZ66FoWizNq-sy8Mk1bf-CMldjijcx9_-AZCnPjC1qRxXu4/s1290/not_correct_copy_docker.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1188" data-original-width="1290" height="377" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjS28ZZd0k2to92uY6LB134TeEhk6Nf8wparbxqMTNZHlqMs_sXVDKF6IS5xymuO6lPsK6yQkMm-ybskF5YFOB7oXnvIvgHqPZ66FoWizNq-sy8Mk1bf-CMldjijcx9_-AZCnPjC1qRxXu4/w410-h377/not_correct_copy_docker.png" width="410" /></a></div><br /></span></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on">Instead of copying all files, copy only dependency file (package.json for NodeJS or requirements.txt for python), install dependencies and then copy all source files.</div><div dir="ltr" trbidi="on"><span style="caret-color: rgb(255, 0, 0); color: red;"><br /></span></div><div dir="ltr" trbidi="on"><span style="caret-color: rgb(255, 0, 0); color: red;">CORRECT WAY:<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDJJwBZVNT4IfpPyaWZJJzCUVQN-BCD2PTuRfUJQ6Se9u3XpI7QQpH_I8X4zyOl4rXKZEDVB7lVL5Oa8-SDqFC31ac2zE2ZG_p3F5onTNG-eI5DglwklpbP75z4ThQ3BcP0VDiRgRiDanU/s1348/correct_copy_docker.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1326" data-original-width="1348" height="403" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDJJwBZVNT4IfpPyaWZJJzCUVQN-BCD2PTuRfUJQ6Se9u3XpI7QQpH_I8X4zyOl4rXKZEDVB7lVL5Oa8-SDqFC31ac2zE2ZG_p3F5onTNG-eI5DglwklpbP75z4ThQ3BcP0VDiRgRiDanU/w410-h403/correct_copy_docker.png" width="410" /></a></div></span></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b>3. Using RUN stead of CMD:</b> Never use RUN command in Dockerfile, Dockefile is for creating a docker image and not executing any commands. Instead of RUN, use CMD.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8TfwnD3BH2VFz7AR7IwbqNcToIIXABran-nYCVjIap4wsXEf5CzNy7SDRqgCdbJXdGPPAiAQLfdSXlTSme3LsYrOI1yDriHb7EhvGM3mDNvr4kRbLkCS8bgGutRa5HPTF990lfZK-mg1o/s1722/RUN_CMD_docker.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1076" data-original-width="1722" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8TfwnD3BH2VFz7AR7IwbqNcToIIXABran-nYCVjIap4wsXEf5CzNy7SDRqgCdbJXdGPPAiAQLfdSXlTSme3LsYrOI1yDriHb7EhvGM3mDNvr4kRbLkCS8bgGutRa5HPTF990lfZK-mg1o/w513-h320/RUN_CMD_docker.png" width="513" /></a></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b>4. Showing docker containers (running docker images):</b> Most of the docker commands are similar to basic unix commands. We can use unix `ps` command to display all running docker images.</div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b>To display docker images: `docker images`</b></div><div dir="ltr" trbidi="on"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCwEZaThF260APP4bznLw2m1WS8dDr7bvwg5TWABlZR3GQ6rrtatYSE4ooVAkF6mLskS1vbIWjQq-Iuc7IXNsMVJ1_p3GJw5IGlHdqiCl5xiXiPBqODJHOVzJn84BHml3IOll6A11Frw2f/s1698/dcoker_images.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="184" data-original-width="1698" height="55" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCwEZaThF260APP4bznLw2m1WS8dDr7bvwg5TWABlZR3GQ6rrtatYSE4ooVAkF6mLskS1vbIWjQq-Iuc7IXNsMVJ1_p3GJw5IGlHdqiCl5xiXiPBqODJHOVzJn84BHml3IOll6A11Frw2f/w512-h55/dcoker_images.png" width="512" /></a></div><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b>To display docker containers: `docker ps`</b></div><div dir="ltr" trbidi="on"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUzTFn-V8NCT45OAYDyfKS7G_S8hoV9N9wWmUvUg1pMq8fftDPfnHdfB_GFjfBmwBeoRI9RCWjeB-MUyjg7A7fKh9S9B3iVW2FwwrFTwam4wKzSihfkuD7F8VNge5UtyhUsGtFkBQvURV6/s1818/docker_ps.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="204" data-original-width="1818" height="58" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUzTFn-V8NCT45OAYDyfKS7G_S8hoV9N9wWmUvUg1pMq8fftDPfnHdfB_GFjfBmwBeoRI9RCWjeB-MUyjg7A7fKh9S9B3iVW2FwwrFTwam4wKzSihfkuD7F8VNge5UtyhUsGtFkBQvURV6/w512-h58/docker_ps.png" width="512" /></a></div><b><br /></b></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b>5. Use docker exec to run commands on docker container:</b> Using docker exec command, we can run all basic unix commands like `ls`, `cd` etc on docker container while running the docker image. This will be very useful for checking logs in the container. </div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLenJeF1mL1IGaahhSkTMXEtjpJ83tDtL8ERafKE04-2DKd6ymaSeS2K5hgqrQ9h2XYfwc_MAmvDzf3r7jBfwRlvjGBetAK-TXjns2Cxgg0l8Zb3AHGa-O2IZ3xcYsna6msiHsVKTQ3fGx/s1882/dokcer_exec.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="624" data-original-width="1882" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLenJeF1mL1IGaahhSkTMXEtjpJ83tDtL8ERafKE04-2DKd6ymaSeS2K5hgqrQ9h2XYfwc_MAmvDzf3r7jBfwRlvjGBetAK-TXjns2Cxgg0l8Zb3AHGa-O2IZ3xcYsna6msiHsVKTQ3fGx/w512-h170/dokcer_exec.png" width="512" /></a></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b>6. Connect to the docker container:</b> Get the current docker containers by running docker ps command. After that use exec command to connect to docker container shell prompt.</div><div dir="ltr" trbidi="on">`docker exec -it CONTAINERID /bin/bash`</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJSlAQ0iT-frkHIAX8bIzYEpF9s9QpbuNB4IiwugJqBrLYtr1N4qd4NarGhdXMHA1hBEReMybWV_QHzxBx_4IafTTSOYZEribHw28FAmLqXpKYiGO1rvG6RGgwGapT9otMcdGYTNyNr5lC/s1910/docker_exec_container.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="604" data-original-width="1910" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJSlAQ0iT-frkHIAX8bIzYEpF9s9QpbuNB4IiwugJqBrLYtr1N4qd4NarGhdXMHA1hBEReMybWV_QHzxBx_4IafTTSOYZEribHw28FAmLqXpKYiGO1rvG6RGgwGapT9otMcdGYTNyNr5lC/w512-h162/docker_exec_container.png" width="512" /></a></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b><br /></b></div><div dir="ltr" trbidi="on"><b>7. How to access host application from docker container: </b>I spent more time compare with other topics in docker to know this. When running docker in the host and trying to access some endpoints like database from host, docker doesn't know which is host and which is docker. Need to specify the host specifically. If you want to access endpoints from host, you need to specify `<b>host.docker.internal</b>` instead of localhost or 127.0.0.1. This differentiates docker localhost and actual localhost.</div><div dir="ltr" trbidi="on"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXLVj0GEaB8nrhdU7HrUGhSCwTo4zIS2UDzzAV1BLaWq-nHIfue98XTivRQsbWAfTvNyntxFhtvpIcDu8ymC1wk5KUWdO667JtJhd9CKyhTzwSxpfxEuWePusr2Ek7fFxuP-G-oaHo5BI9/s1280/docker_host_access.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="106" data-original-width="1280" height="42" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXLVj0GEaB8nrhdU7HrUGhSCwTo4zIS2UDzzAV1BLaWq-nHIfue98XTivRQsbWAfTvNyntxFhtvpIcDu8ymC1wk5KUWdO667JtJhd9CKyhTzwSxpfxEuWePusr2Ek7fFxuP-G-oaHo5BI9/w512-h42/docker_host_access.png" width="512" /></a></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b>8. Expose Port numbers:</b> When containerizing any API end points, we need to explicitly expose port numbers in the Dockerfile. Otherwise, These ports are not accessible from host machine.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7Rwh7F6LDgvKMnrOFH85_THM5R7WM7Qb1Z6sELgmhnk5WFwfTLl2_6dzKiJ25Q_Z3-Son_lCcSrATvgU6GZNx4w07oSWd3kjvKrp02CdRFu9REAuF9D8Y5Ne7JmoKZhU9_VmUk1uFmD69/s1050/docker_expose.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1050" data-original-width="778" height="410" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7Rwh7F6LDgvKMnrOFH85_THM5R7WM7Qb1Z6sELgmhnk5WFwfTLl2_6dzKiJ25Q_Z3-Son_lCcSrATvgU6GZNx4w07oSWd3kjvKrp02CdRFu9REAuF9D8Y5Ne7JmoKZhU9_VmUk1uFmD69/w303-h410/docker_expose.png" width="303" /></a></div><div class="separator" style="clear: both; text-align: left;"><b>Run docker image:</b> Run docker image by specifying exposed ports in the run command as below.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgAJ12c8QvE8i_vGa8Yaol73BlCd32ocQu6BoKAbFFEwvO_Cm9ATwylzL-Wl9TdZzKlLweL72h0M6jrB3R4q4ucLAc0x3x63SbbZF89hXAAUT4emzPTbYLyaZF9vetpVQTVWoaB_kBGMPl/s1716/docker_run.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="88" data-original-width="1716" height="26" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgAJ12c8QvE8i_vGa8Yaol73BlCd32ocQu6BoKAbFFEwvO_Cm9ATwylzL-Wl9TdZzKlLweL72h0M6jrB3R4q4ucLAc0x3x63SbbZF89hXAAUT4emzPTbYLyaZF9vetpVQTVWoaB_kBGMPl/w512-h26/docker_run.png" width="512" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Happy Learning!!!!</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div dir="ltr" trbidi="on"><br /></div><div><br /></div></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-14198495838984096592020-04-30T10:48:00.001+05:302020-06-19T12:22:44.587+05:30Most frequently used methods of Pandas data frame in Python!!!<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Recently I worked on a machine learning project <b><a href="https://github.com/ageitgey/face_recognition">Face Recognition</a>.</b> Basically this project will take one image as input and return list of images which contains similar faces from input image. For this I used python framework <b>pandas</b> <span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span> for preprocessing images to identify good face in the image. Below are some of the data frame operations I used.<br />
<br />
<strong>1. Create a empty </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>2. Create a </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><strong> from a list</strong><br />
<strong>3. Length of the </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>4. Sorting data frame basing on specific column</strong><br />
<strong>5. Extract a value from </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>6. Iterate through </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>7. Filter/Retrieve </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><strong> basing on column value</strong><br />
<strong>8. Extract some columns from </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>9. Reading a column values (Unique) as list from </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>10. Unique count from a column in </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>11. Adding index column to the </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>12. Delete a column from </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>13. Remove duplicates from </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>14. Save </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><strong> as a csv file</strong><br />
<strong>15. Read csv file as </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><br />
<strong>16. Merge two </strong><b>DataFrames</b></div><div dir="ltr" style="text-align: left;" trbidi="on"><strong>17. </strong><strong>Updating Dataframe column</strong></div><div dir="ltr" style="text-align: left;" trbidi="on"><strong>18. </strong><strong>Splitting </strong><span class="ILfuVd"><span class="e24Kjd"><b>DataFrame</b></span></span><strong> column</strong><br />
<br />
Check GitHub <a href="https://github.com/chanduthedev/ML/blob/master/pandas-dataframe-utils.ipynb">link</a> for Jupyter notebook code for above operations.<br />
<br />
Happy Coding!!<br />
Happy Learning!!</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0Singapore1.352083 103.8198360.8441055 103.17438899999999 1.8600605 104.465283tag:blogger.com,1999:blog-2374446609565222827.post-76795011857315177172019-10-05T20:54:00.001+05:302019-10-05T20:54:05.327+05:30Useful commands for data cleaning in Machine Learning(ML) training - part 2<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
This post details post of all the commands specified in <a href="http://www.chanduthedev.com/2019/09/useful-commands-for-data-cleaning-in.html">this</a> post.<b> </b><br />
<b><br /></b>
<b>grep:</b> If you are regular CLI user, you might have already knew it about grep command and its advantages. You can refer <a href="http://www.chanduthedev.com/2012/03/grep-command-in-unix.html">this</a> and <a href="http://www.chanduthedev.com/2019/03/grep-unix-command-useful-tips.html">this</a> for more info on grep.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span><br />
<b>pipe(|):</b> Pipe is used to combined more than one command. i.e output of one command is used as a input for another command. This will help to make multiple commands in a single line. If you want to find list of names from a file and sort them by removing duplicates. You can simply do like this <span style="color: blue;"><i>'grep name file_name | sort -u'</i></span><br />
<br />
<b>cat:</b> Cat is the command to show or display contents of the file in the console or stdout. Earlier I used vim for seeing content of the file. But now I am using cat along with more command to see the contents. In my opinion this is the best way to see contents. And if you want to process file content, you can use cat along with awk.<br />
<br />
<b>awk:</b> This is very power full text processing tool. We can do text processing efficiently and effectively using this command. Input data for training should be always separated by delimiters like comma(,), pipe(|) or hyphen(-) or mostly CSV format. We can use awk to verify whether input file is formatted properly. You can refer awk use <a href="http://www.chanduthedev.com/2019/09/how-to-validate-data-file-is-formatted.html">here</a>.<br />
<br />
<b>> (redirection):</b> This will help to save intermediate files in a separate file. For example, if you want to save sorted names after searching, you can run this command <span style="color: blue;"><i>'grep name file_name | sort -u > new_filename</i></span>'. This will save results in a file new_filename.<br />
<br />
<b>wc :</b> This command will show total no.of lines, words and characters in a file. I never used this command even though I knew it until I started working for ML project. If you are working on large size files like machine learning projects where required input data is large in size, it is always good verify the file size or no.of lines. In that case this command will really helps a lot. You can find total no.of unique names after grep by combining this command with grep and sort as <span style="color: blue;"><i>'grep name file_name | sort -u| wc'</i></span>. If you want to know only lines or characters or words just use <i><span style="color: blue;">'wc -l filename'</span></i> for total no.of lines from filename, for words use 'w' and 'c' for characters instead of 'l'.<br />
<br />
<b>more:</b> This command is used to see the file content page by page if the content is more than one page. This will really help in seeing and analyzing the content of the file very quickly. And we can also use this command to see the results of a another command by pipe to analyze the results. For example, I want to see uniquely sorted names after grep, I can use like <span style="color: blue;"><i>'grep name file_name | sort -u | more'</i></span> which will show results in page by page format.<br />
<br />
<b>head/tail:</b> Some times its always helpful to see first or last lines of the file. As the name says, 'head filename' will show first ten lines and <span style="color: blue;"><i>'tail filename'</i></span> will show last 10 lines of the file by default. If you want only first or last three liens you can use 'head -3 filename' or 'tail -3 filename'. You can replace '3' with any numeric if you want. You can also use this with other commands using pipe(|) like <span style="color: blue;"><i>'grep name file_name | sort -u| head'</i></span> or <span style="color: blue;"><i>'grep name file_name | sort -u | tail'</i></span><br />
<br />
<b>file:</b> <span style="color: blue;"><i>'file file_name'</i></span> command will tell the format of the file file_name. Some times filename doesn't contains extension or may be file contains incorrect file extension. In that case you can use this command to know the file format. This is my favorite command. I use this regularly to know file format. One of my personal experience was when I started working in iOS development. I really don't know about 'ipa' file format. I learned by using this file command that ipa is a zip file. You can see related post <a href="http://www.chanduthedev.com/2013/08/what-is-ipa-file-how-to-open-ipa-file.html">here</a>. If you are working in machine learning project where file size going to be large, you cant open the file. This command will help in that scenario.<br />
<br />
<b>ls -h:</b> I know many of you aware of this command and you might have already used <span style="color: blue;"><i>'ls -ltr'</i></span> many times. But I am going tell one usefull option is reading the file size quickly. By using '-h' option, ls command will disply the file size in human readable format. This is also one of my favorite command and it is very very help full when you are working CLI.<br />
<br />
Happy Learning!!!</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com2tag:blogger.com,1999:blog-2374446609565222827.post-42615356337568068942019-09-29T18:46:00.002+05:302019-09-29T18:46:56.318+05:30Key points to better understand Machine Learning(ML) projects!!!!!<div dir="ltr" style="text-align: left;" trbidi="on">
<b><br /></b>
<b>Problem statement:</b> Understanding problem statement is one of the key to get good Machine learning model. Domain knowledge makes crucial role in understanding problem statement.Defining problem statement is not easy. All the examples or problems available over the web are clearly defined their problem statement. But when you start working on real time use cases, its very hard to understand the problem. Data scientist and Data Analyst will play major in this.<br />
<b><br /></b>
<b>Data: </b>Now a days we are having lot of data in the form of text, images, audio and video format. But the problem with this data is, all this data is unstructured format and not clean.All the sample data available over web (<a href="https://www.kaggle.com/">Kaggle </a>for example) is already cleaned data. For practicing or for learning this will help. But when you started working on real time projects, you wont get cleaned data. Data Engineer will play major role in cleaning unstructured data, which is commonly known as <b>pre-training</b> stage.<br />
<br />
<b>Understanding Data: </b>Even though cleaned data is available, To get better training model, need to understand data samples clearly. How data is a distributed and what features needs to take from that data.If the data is not distributed equally, ML model will not work properly. Always make sure that input data is equally distributed. Data Analyst will play major role in this.<br />
<br />
Happy Learning!!</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com1tag:blogger.com,1999:blog-2374446609565222827.post-19006557476798769092019-09-22T09:55:00.000+05:302019-10-05T20:46:08.083+05:30Useful commands for data cleaning in Machine Learning(ML) training - Part 1<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
If you are a CLI user , some of these commands may be familiar to you. The hardest part in whole Machine learning(ML) is providing clean data to train. ML is all about data and there is lot of data available these days and huge data is generating every data in the form of text, videos, images etc. All this data is not structured and that is one of the biggest problem. How to make unstructured into a usable format? By using combination of below commands, we can clean raw data by removing noise and make a structured data which further can be used for training a model.<br />
<br />
<ul style="text-align: left;">
<li>grep</li>
<li>pipe(|)</li>
<li>cat</li>
<li>awk</li>
<li>> (redirection)</li>
<li>wc -l</li>
<li>more</li>
<li>head</li>
<li>tail</li>
<li>ls -lh</li>
<li>sed</li>
</ul>
<br />
<b>grep:</b> is a command to find or search for a particular pattern in a given file. you can find more on this.<br />
<b>pipe(|):</b> pipe alone we cant use. But this will be used to join more than one command.<br />
<b>cat:</b> is a command to show contents of a given file in the console.<br />
<b>awk:</b> is very powerful command line utility for text processing.<br />
<b>>(redirection):</b> is the utility to redirect results of any command from console to any other file<br />
<b>wc:</b> This command will give total lines, words and characters from a given file. -l will give only total lines from a given file. This will be very useful to know how big the data file is.<br />
<b>more:</b> this is to see the contents of a given file in pagination format if file content is more than one page. Initially I used to use vi to see the contents of a file, after knowing about more command, I started using more instead of vi and it is very effective as well.<br />
<b>head:</b> to see sample content(first 10 lines) from a given file. you can use -n to specify how many lines you want to show on the console<br />
<b>tail:</b> same as head, but from bottom. To see sample content(last 10 lines) from a given file. you can use -n to specify how many lines you want to show from bottom on the console<br />
<b>ls -lh:</b> Many of us know about ls command. trick here is l and h. l will show file details like permissions and owner etc, h will show the size of the file in human readable format. like if file size is 1424, ls -h will show 1.4K. you can easily tell later one is more readable.<br />
<b>sed:</b> is a stream editor. this command mostly I used for replacing a pattern in the same file.<br />
<div>
<br /></div>
I write next post on these commands in details.<br />
<br />
Happy Learning!!!<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-72989563656489135142019-09-02T19:21:00.001+05:302019-09-02T19:22:47.700+05:30How to validate data file is formatted properly before training ML model!!<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
In <b>machine learning</b>(ML), clean data for training is the key for a better model.<br />
Must remove noise(unwanted data like special characters) from input data to get clean clean data.<br />
.To do this one of the first step is to make sure data format is consistent in entire input file.<br />
One of the best way to check this is below command. This command should return<br />
Only one value, if not your data file is not properly formatted.<br />
<br />
<b>cat file_filename | awk -F’,’ ‘{print NF}’ | sort -u</b><br />
<br />
Let’s see in detail about this command. This command contains below sub parts<br />
<br />
<b>cat:</b> is the command to display contents of a text file<br />
<b>awk:</b> this is very power full text processing tool.<br />
<b>sort:</b>this command is to sort and u is for unique and removing duplicates<br />
<b>NF:</b> is number of fields. This will be used In <b>AWK</b><br />
<b>Pipe(|):</b> this is also very power full command line utilitiy which combines more than one command.<br />
<br />
Let’s take a sample file which contains name, age and city of some customers separated by comma, which means each line should contain only three fields separated by comma. But if you see in the below data, in the second line it contains four fields separated by comma.This is very very common scenario in Machine learning training data. And checking this format in a large file not easy. Using above command we can easily verify it.<br />
<br />
Chandra, 35,Singapore<br />
Sekhar,Chandra, 26,Guntur<br />
Sekhar,35,Singapore<br />
<br />
Applying above command on this text will return the results 3 and 4 as shown below as input data is not consistent. This is due to second line contains one extra comma in it.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCsZZ7KzJAb3cb78kyl1FYPsa0NjzsKLYC_3e70KaRj3URg9HxUdvx3KvEG85nn52ZihzP82Mxu1zfnaU1RK3-HZtwJmEFGcqhyxlz1Ks7M9smnvtKVGhXHyY4nEVZPEcg7W_ErEvdK7L2/s1600/NF.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="151" data-original-width="369" height="130" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCsZZ7KzJAb3cb78kyl1FYPsa0NjzsKLYC_3e70KaRj3URg9HxUdvx3KvEG85nn52ZihzP82Mxu1zfnaU1RK3-HZtwJmEFGcqhyxlz1Ks7M9smnvtKVGhXHyY4nEVZPEcg7W_ErEvdK7L2/s320/NF.PNG" width="320" /></a></div>
<br />
After removing extra comma from the second line, that command returns only one value which is 3 as shown below. This return value may vary basing on number of fields in each line, but each line should contain same number of fields which makes this command returns only one value.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSVcZgJfsjN2zmfccfhh3Q1tKrGO2TF7J8QI9583aPZR5b418uMrP4LFu6I0mGDXalh-yDlxd27pVmutaDSbq0_SFTytEBRg2PDSvQ7llVMLns-S0Lz4WZKUdy8NsIrDKFjXnNHBGBDL99/s1600/result.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="133" data-original-width="373" height="114" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSVcZgJfsjN2zmfccfhh3Q1tKrGO2TF7J8QI9583aPZR5b418uMrP4LFu6I0mGDXalh-yDlxd27pVmutaDSbq0_SFTytEBRg2PDSvQ7llVMLns-S0Lz4WZKUdy8NsIrDKFjXnNHBGBDL99/s320/result.PNG" width="320" /></a></div>
<br />
<br />
<b>cat </b>command will display the contents of the file and redirect to AWK because of <b>pipe </b>and <b>awk </b>will split each line with comma as -F is a field separator and NF will return all the fields after split each line. In this case we have three lines in that first and third lines has three and second line has 4 after split using comma. Sort will show the sorted result with duplicates and -u will show only unique values.<br />
<br />
This command will work efficiently on very large files as well. I tried on data files whose file size is around one million lines and it returned results in seconds.<br />
<br />
<br />
Happy Learning!!!<br />
<br /></div>
</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-4076357687416018142019-03-12T20:05:00.000+05:302019-03-12T20:05:11.787+05:30GREP unix command useful tips!!!<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I learned useful tips from my manager for '<b>grep</b>' which helps a lot when troubleshooting the issues with logs. In this post I will explain about below<br />
<br />
<ul style="text-align: left;">
<li>How to search multiple strings in a file at the same time.</li>
<li>How to display found string in the color.</li>
<li>How to exclude particular string in the search result</li>
<li>How to display total no.of lines in the grep result</li>
</ul>
<br />
<br />
<b>Search a string in a file using grep: </b>Check this <a href="http://www.chanduthedev.com/2012/03/grep-command-in-unix.html">link</a> for basic usage of grep command. <b> </b></div>
Lets see this tip with example. Below is the content of a file <b>demo.txt </b>and this file name we are going to use in this example as well.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAnxrad0HMMzVOBZvjmzfDrLNB1PaTEPiKFCi3g-Iy-gDwRZjswOI6BWQylbF_P9XbSsaeDlaZbmxXHTU1rpd_Ql9xZ80sXnSdm_f9sRp7-ksbETtFwOMsDwansY_GPm6vnN9vixeFpZGo/s1600/Screenshot+2019-03-11+at+3.17.40+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="242" data-original-width="1540" height="62" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAnxrad0HMMzVOBZvjmzfDrLNB1PaTEPiKFCi3g-Iy-gDwRZjswOI6BWQylbF_P9XbSsaeDlaZbmxXHTU1rpd_Ql9xZ80sXnSdm_f9sRp7-ksbETtFwOMsDwansY_GPm6vnN9vixeFpZGo/s400/Screenshot+2019-03-11+at+3.17.40+PM.png" width="400" /></a></div>
<br />
<br />
<b>How to search multiple strings in a file:</b><br />
In the below screen we are search for the strings 'demo', 'show' and 'multiple' in the file <b>demo.txt </b>and and the result is as follows.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3PZv-uI1fDnjp6HtDQPnjltSG6q_Y-xlBRGaacUYYghWkqSD2zjkUMmGCyzZB8xtiiH28EVEwsIScLPoL0WyIRWwxpyJeCv1ReNZyIwa4ks4lKVvk-BjpVUpcIaSb3M-RoCviDQz2joht/s1600/Screenshot+2019-03-11+at+4.29.25+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="189" data-original-width="1600" height="46" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3PZv-uI1fDnjp6HtDQPnjltSG6q_Y-xlBRGaacUYYghWkqSD2zjkUMmGCyzZB8xtiiH28EVEwsIScLPoL0WyIRWwxpyJeCv1ReNZyIwa4ks4lKVvk-BjpVUpcIaSb3M-RoCviDQz2joht/s400/Screenshot+2019-03-11+at+4.29.25+PM.png" width="400" /></a></div>
<b>How to display found string in color:</b><br />
In the above screen, even though grep found the patterns, it is difficult to identify in which line these patterns available. For that you can use grep command property '<b>color</b>' as shown below.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7hzmY88nH9a8dS_jgX2FA2TQQBLypczjTiy3j_vS0fr7YgKKTmR_nbWOFylDemxhvv1hMmrLD_mziLDK6tJllCBwGQf7-E9QQMoMyTHj3vFY9kv7PbDM93gQFReIsyirhSM6wAc3RMsNC/s1600/Screenshot+2019-03-11+at+4.31.58+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="167" data-original-width="1600" height="41" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7hzmY88nH9a8dS_jgX2FA2TQQBLypczjTiy3j_vS0fr7YgKKTmR_nbWOFylDemxhvv1hMmrLD_mziLDK6tJllCBwGQf7-E9QQMoMyTHj3vFY9kv7PbDM93gQFReIsyirhSM6wAc3RMsNC/s400/Screenshot+2019-03-11+at+4.31.58+PM.png" width="400" /></a></div>
<br />
From the above screen, we can easily identify the found strings as they are highlighted in red color. This color utility will save lot of time when you are searching in debug logs while troubleshooting.<br />
<br />
<b>How to exclude particular string in the search result: GREP </b>command will supports option to exclude particular pattern from the result. This is basically not including a string in the result.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifuc5As4KhnqsFZNCrpTrrFCpBiKBZkQHbaketp2an5SiXoSpMtA8l001Au2rEqHVIMBdzQ_lP4fZp8UX6Dseyhz6F8uwjZhmsfZ7DDwnyCirjgNHwWpqQjgE4tZu4zltUEo3wuR013dz5/s1600/Screenshot+2019-03-12+at+9.20.06+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="119" data-original-width="1600" height="28" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifuc5As4KhnqsFZNCrpTrrFCpBiKBZkQHbaketp2an5SiXoSpMtA8l001Au2rEqHVIMBdzQ_lP4fZp8UX6Dseyhz6F8uwjZhmsfZ7DDwnyCirjgNHwWpqQjgE4tZu4zltUEo3wuR013dz5/s400/Screenshot+2019-03-12+at+9.20.06+PM.png" width="400" /></a></div>
<br />
<br />
In the above screen, initially search for the strings 'demo' and 'show' and in the results I want to exclude the string 'multiple'. <br />
<br />
<b>How to display total no.of lines in the grep result: </b>We can use <b>-c </b>option to get the total number of lines of the grep result. If the pattern is unique in each line, that count will be total no.of occurrences of the pattern.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm89VYH1Nq62SbcG4XzpJ4ukaTixZFQWc0JgILq1U1VyDJ3_s_LNNannaKwQaxCg24-BUC9R8PRsbSf7xM0SEoSph2ggIqj_E0nggKn4Z6yTo-htEbahFu53tf6hfqZew3RDsgKwPVPLpB/s1600/Screenshot+2019-03-12+at+9.25.33+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="478" data-original-width="1600" height="118" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm89VYH1Nq62SbcG4XzpJ4ukaTixZFQWc0JgILq1U1VyDJ3_s_LNNannaKwQaxCg24-BUC9R8PRsbSf7xM0SEoSph2ggIqj_E0nggKn4Z6yTo-htEbahFu53tf6hfqZew3RDsgKwPVPLpB/s400/Screenshot+2019-03-12+at+9.25.33+PM.png" width="400" /></a></div>
<br />
<br />
In the above screen, searching for the patterns 'demo' and 'show' results two lines and using <b>-c </b>option will show the total count to two. If the grep results are more, this count option will be very useful.<br />
<br />
<br />
Happy Learning!!!<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-84607787555203508662019-02-24T19:58:00.000+05:302019-02-25T15:09:14.450+05:30Uploading a file using Multer!!<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="margin-left: 1em; margin-right: 1em;">
</div>
<br />
Recently I got a requirement of uploading a file. We are using NodeJS server, so I explored on NPM and found out very simple library called <b>Multer</b>. It is very simple to use, so sharing here.<br />
For uploading a file, we need to know the location where we are going to store the file. For that we need to specify the location for the <b>Multer</b> as shown below. One parameter is <b>destination</b> in the line 20 where we need to specify the location in this case <b>uploads </b>folder in the current directory. Another parameter is <b>filename</b> un the line 23 with which we are going to store in the location. <span style="color: red;">If filename not specified, random number will be given to that file name. </span><br />
After specifying destination and filename parameters, need to map these <b>Multer</b> storage fields to <b>Multer</b> as shown below in the line 29.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhetJyPh-WQJeZ1QTc5gSDRyQ50ogY-A9luAoPmhYB0RGGhj5hph2r3mem8QzhhPdZYMtkf948MBLTltJrVF1Wuhfbs9tVJba9pu3mQqrEhqQsMar4LVxbiF1w3oiYIyyBh16TtYWlG6IzB/s1600/Screenshot+2019-02-24+at+9.50.02+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="590" data-original-width="1048" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhetJyPh-WQJeZ1QTc5gSDRyQ50ogY-A9luAoPmhYB0RGGhj5hph2r3mem8QzhhPdZYMtkf948MBLTltJrVF1Wuhfbs9tVJba9pu3mQqrEhqQsMar4LVxbiF1w3oiYIyyBh16TtYWlG6IzB/s400/Screenshot+2019-02-24+at+9.50.02+PM.png" width="400" /></a></div>
<br />
Now let's see how to upload file. While using uploading POST request, need to match the query which is coming in the request parameters as shown below in line 48. In this api , I am using '<b>file</b>', so query parameter should be '<b>file</b>', you can use any name, but i used file here. In the line 48, we are using <b>upload.single('file'), </b>upload is actually <b>Multer</b> configured storage in the above image line 29. If file uploading is successful, we will get file name in the call back as shown in the line 49. If file upload fails, this filename will be undefined.<br />
<br />
If you want to validate the request before start uploading, just add the method in the POST request as shown below in the line 48. In this case I used a method name as <b>validate, </b>you can use whatever function name you want or even you can skip this validation if you don't want, in that case you just remove that validation method from the line 48.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4Brc8hS1dgN3-8KINIsGAlDkgDBXLXHr2W4WE2F8tZoVUkZQ7DDAeLaI_zFYei9iPB2khlgxlk4FpdfvS-Yg8Jd0euFMHNvZeC9Y73-uExEI_Ln4lWumyatm7QkGG08_K-xHjH0aIoeDb/s1600/Screenshot+2019-02-24+at+10.24.45+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="611" data-original-width="1600" height="152" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4Brc8hS1dgN3-8KINIsGAlDkgDBXLXHr2W4WE2F8tZoVUkZQ7DDAeLaI_zFYei9iPB2khlgxlk4FpdfvS-Yg8Jd0euFMHNvZeC9Y73-uExEI_Ln4lWumyatm7QkGG08_K-xHjH0aIoeDb/s400/Screenshot+2019-02-24+at+10.24.45+PM.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
Now let's see Downloading file. For downloading a file, there is no dependency with Multer, but as we are discussing about uploading a file, lets see downloading file as well.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0bijSor5vKb-7BwJlrWXIMJoVxv6FA0vX_vhAg6dW5BzMSAFfnYT0IJMCfy37IlwTDpEQCprBQeKK0pkPZlOxxRblyZskQFOygyhobpVCzjsuv5b1N6gS0XxZFVZTjW45b89WAuZJ8HdV/s1600/Screenshot+2019-02-24+at+10.12.11+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="788" data-original-width="1596" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0bijSor5vKb-7BwJlrWXIMJoVxv6FA0vX_vhAg6dW5BzMSAFfnYT0IJMCfy37IlwTDpEQCprBQeKK0pkPZlOxxRblyZskQFOygyhobpVCzjsuv5b1N6gS0XxZFVZTjW45b89WAuZJ8HdV/s400/Screenshot+2019-02-24+at+10.12.11+PM.png" width="400" /></a></div>
<br />
For the complete code. Refer my github link <a href="https://github.com/chanduthedev/nodejs/tree/master/multer">here</a>.<br />
<br />
Happy Learning!!<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-49503087005640182142017-07-06T17:08:00.002+05:302017-07-06T17:24:59.439+05:30What is functional programming?<div dir="ltr" style="text-align: left;" trbidi="on">
<b> Functional programming</b> (<b>FP</b>)is one of the programming style like procedural or object oriented programming. It will follow <b>Declarative</b> programming paradigm in which, programming will be done using expressions instead of statements. Where as <b>Imperative</b> programming language uses statements(e.g C, Java). Below are the some of the functional programing concepts.<br />
<ul style="text-align: left;">
<li>Pure functions</li>
<li>No Side effects</li>
<li>No shared state</li>
<li>No Mutating state</li>
<li>Function composition </li>
</ul>
<b>Pure Functions: </b>A pure function is a function which will return always same value for the same input. The result of a pure function is always depends on input values and its internal logic. Pure function will not read/write anything from outside of that function. A pure function will eliminate side effects and maintains <b>referential transparency</b>. A pure function can be referential transparency if function call can be replaced with its result. Below are some of the examples.<br />
<ul style="text-align: left;">
<li>'chandu'.length is always 6, so its pure function. And instead of 'chandu'.length, you can use 6 as well, so this is also a referential transparency.</li>
<li>Max(3,7) is always 7, so Max is a pure function</li>
<li>sqrt(9) is always 3, so its a pure function. </li>
</ul>
<b>No Side effects: </b>As stated in pure functions, there will not be any out side change due to a particular function call. Should avoid shared variables, global variables to eliminate side effects.<br />
<br />
<b>No Shared State: </b>Shared state is sharing variable or object or memory location. In functional programming there will not be any shared variable or object. If shared state is there, it violates pure function definition as some other function may change shared object. If we eliminate shared state, there wont be any change in the order of the function calls as well.<br />
<br />
<b>No Mutating state: </b>A mutable object can be modifiable after its creation. In functional program, modification of an object is not allowed after its creation. So that there wont be any side effects. SO Immutability is the another feature of functional programming.<br />
<br />
<b>Function composition: </b>Composition is a combination of two or more functions to make a new function. As stated earlier Functional programming is declarative language, so it uses expressions instead of statements. This function composition is very useful to make expressions using functions.<br />
<br />
Functional programming languages are used mostly on mathematical calculations and pattern matching and AI. Haskell and Lisp are some of the FP languages. JavaScript and Python will follow some of the FP features. <br />
<br />
Happy Learning!! <a href="https://medium.com/javascript-scene/master-the-javascript-interview-what-is-functional-programming-7f218c68b3a0">References</a><br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com3tag:blogger.com,1999:blog-2374446609565222827.post-88592399433142271942017-03-30T16:32:00.002+05:302017-03-30T16:32:50.389+05:30What is Apple file system(APFS)!!!<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Apple announced its new file system called <b>Apple File System(APFS) </b>at <a href="https://developer.apple.com/videos/play/wwdc2016/701/">WWDC2016</a>. Lets see what are new features in APFS and why apple introduced this new file system.<br />
<br />
Till now apple is using file <b>HFS+(</b>Hierarchal file system<b>)</b>. The problem with this file system is, it designed almost three decades back and it designed basing on the existing drives(HDD's, FlopDisks) on that time. Though HFS+ supports latest SSD, Flash drives, it is not efficient enough. To support all these, Apple introduced new file system called <b>Apple File System</b>. Below are the some of the key features of <b>APFS</b>.<br />
<br />
<ul style="text-align: left;">
<li>To make single file system for all apple OS (macOS, iOS, watchOS and tvOS)</li>
<li>To support and take advantage of SDD and Flash memory disks</li>
<li>64-bit support </li>
<li>Encryption first</li>
<li>Space sharing- if partition the disk, this file system will automatically adjusts the memory if one partition has no space</li>
<li>Feel fast access by reducing the latency - as SSD and flash disks wont have any spinning needle to read/write unlike HDD's.</li>
<li>Snap shot and clones - copying and moving files are quick</li>
<li>Less OS Space</li>
</ul>
<div>
This file system will be available from iOS 10.3, macOS 10.12.4. When users upgrades to 10.3 it version, it will automatically converts file system from existing HFS+ to new Apple File system. Though this conversion will not effect your data, its advised to <span style="color: red;">take the back up </span>before upgrading iOS 10.3 </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Happy Upgrading!!!</div>
</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-34048613579351248782016-12-06T17:00:00.001+05:302016-12-09T12:49:19.803+05:30What is Optional in Swift!!!<div dir="ltr" style="text-align: left;" trbidi="on">
Apple's new programming language language <b>swift </b>is a very safe language. It will try to make sure that your code is not going to crash. To do this swift provides a feature called <b>optional </b>type. This optional type will store wrapped value if present and nil if no value presents. That basically means any optional variable either contains a value or nil. Lets see deep into optional type. The Optional type is a enumerated value with two values <b>nil </b>and some value which are represented as below.<br />
<ul style="text-align: left;">
<li>Optional.none</li>
<li>Optional.some(value)</li>
</ul>
<b>How to use optional: </b>There are two ways to use - <b>long form</b> and <b>short form</b>. Mostly short form will be used, but we will see here both for better understanding purpose. Short form is represented by post question mark ?, and long form is represented by <b>Optional </b>key word.<br />
<br />
<pre class="brush: js">let shortForm: Int? = Int("77")
let longForm: Optional<int> = Int("77")</int>
</pre>
<br />
<b>Optional Binding: </b>We can use optional variable by unwrapping the value, so that there wont be any runtime error. To unwrap conditionally we have three options<br />
<ul style="text-align: left;">
<li><b>if let</b></li>
<li><b>gaurd let</b></li>
<li><b>switch</b></li>
</ul>
<b>Optional Chaining:</b> Optional chaining is a process for calling methods and properties on a optional that could be nil. if the optional contains a value, it will succeeds and proceeds for the next value, if the optional is nil, it simply returns nil. multiple optional method callings can be chained together and the entire chain will be failed gracefully if any one of the value is nil.<br />
<br />
<pre class="brush: js">if let someResult = someValue?.someMethod()?.someAnotherMethod()
{
print("Success")
} else {
print("failed")
}
</pre>
<b>Nil-Coalescing Operator:</b> This will be used for the optional to set default value for the nil optional value. And this can be used by doubel question mark ??. This can be also used as chaining<br />
<br />
<pre class="brush: js">let someResult = someValue?? anotherValue</pre>
<br />
In this case, <i>someResult </i>will be <i>someValue </i>if it has value and <i>anotherValue </i>if <i>someValue </i>is nil.<br />
<br />
<b>Chaining example:</b><br />
<pre class="brush: js">let someResult = someValue?? anotherValue?? anotherValue1
</pre>
<b><br /></b>
<b>Unconditional Unwrapping:</b> If you are sure that optional has a value,then this unconditional unwrapping will be used by specifying forced unwrap operator (postfix !). The problem with this feature is, if the optional value is nil, you will get run time error or possibly your app may crash.<br />
<br />
<pre class="brush: js">let number = Int("77")!
print(number)
// Prints "77"
</pre>
<br />
It is also possible to use chaining using postfix !.
<br />
<pre class="brush: js"></pre>
<pre class="brush: js">let isPNG = imagePaths["image"]!.hasSuffix(".png")
print(isPNG)
// Prints "true"
</pre>
<br />
Happy <span style="color: blue;">Swift</span>ing!!!<br />
<br />
References:<br />
<a href="https://developer.apple.com/reference/swift/optional">Apple Doc</a><br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-91397512482388954162016-12-05T15:08:00.003+05:302016-12-05T15:08:52.718+05:30Difference between class and struct in Swift!!!<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Structures and classes in swift follow the same syntax for variable, constants and methods. And below are the common things in both class and struct.<br />
<ul style="text-align: left;">
<li>Properties</li>
<li>Methods</li>
<li>Initializers to initialize values</li>
<li>Conform to protocols</li>
</ul>
And the main <span style="background-color: white;"><span style="color: blue;">differences </span></span>are<br />
<ul style="text-align: left;">
<li>Class supports inheritance and structs not</li>
<li>Classes are reference types and structs are value types</li>
</ul>
<div>
<b>Value Type: </b>When copying or assigning one value type variable to another value type variable, whole data will be copied and they are entirely two new objects. If any change in one object doesn't effect another object. <b>Structures </b>are value types.<br />
<br /></div>
<pre class="brush:cpp">//struct definition
struct Name{
var first = "Pasumarthi"
var last = "Chandra"
}
//struct variable initialization
var name = Name()
//copying name objet to anotherName object
var anotherName = name
//modifying
anotherName.last = "Chandra sekhar"
print("\(name.first) \(name.last)")
print("\(anotherName.first) \(anotherName.last)")
Result:
Pasumarthi Chandra
Pasumarthi Chandra sekhar
</pre>
<br />
In the above swift code snippet, I have created structure object <i><span style="color: #cc0000;">name</span> </i>and which is assigned to another structure object <i><span style="color: #cc0000;">anotherName </span></i>and modified <i><span style="color: #cc0000;">anotherName </span></i>property <i><span style="color: #cc0000;">last</span>, </i>and displayed both <i><span style="color: #cc0000;">name </span></i>and <i><span style="color: #cc0000;">anotherName </span></i>objects and both displayed different values.<br />
<div>
<br /></div>
<div>
<b>Reference Type: </b>When copying or assigning one reference type variable to another reference type variable, both will be pointing to the same object and if any change in one object will effect the another object as both are pointing to the same reference. <b>Classes </b>are reference types.
<br />
<br />
<pre class="brush:cpp">//Class definition
class NameAsClass{
var first = "Pasumarthi"
var last = "Chandra"
}
//class initialization
var nameAsClass = NameAsClass()</pre>
<pre class="brush:cpp">//Copying one class object to another class object
var anotherNameAsClass = nameAsClass</pre>
<pre class="brush:cpp">//modifying
anotherNameAsClass.last = "Chandra sekhar"
print("\(nameAsClass.first) \(nameAsClass.last)")
print("\(anotherNameAsClass.first) \(anotherNameAsClass.last)")
Result:
Pasumarthi Chandra sekhar
Pasumarthi Chandra sekhar
</pre>
<br />
In the above swift code snippet, I have created class object <span style="color: #cc0000;">n<i>ameAsClass</i> </span>and which is assigned to another class object <span style="color: #cc0000;"><i>anotherNameAsClass</i> </span>and modified <span style="color: #cc0000;"><i>anotherNameAsClass </i></span>property <i><span style="color: #cc0000;">last</span></i>, and displayed both <span style="color: #cc0000;"><i>nameAsClass</i> </span>and <span style="color: #cc0000;"><i>anotherNameAsClass</i> </span>objects and both displayed same values.</div>
<div>
<br /></div>
References:<br />
<a href="https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/ClassesAndStructures.html#//apple_ref/doc/uid/TP40014097-CH13-XID_97">Click</a><br />
<br />
<br />
Happy Coding!!!!<br />
<br />
<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com1tag:blogger.com,1999:blog-2374446609565222827.post-3364744313028431262016-12-01T16:32:00.001+05:302016-12-02T14:26:57.839+05:30How to write safe and secure code!!!<div dir="ltr" style="text-align: left;" trbidi="on">
Coding is very easy task to do, but writing safe and secure code is difficult. In this post I will try to explain some of the rules/steps to make safe and secure code. If we follow these steps we can eliminate most of the failures in our software. As a coder, we need to find out all possible failure cases first and handle them. Some where I read statement like <b>"A developer is like a cab driver in India who sees both sides in one way road"</b>. I think this statement is very true. In software application anything can happen, no software is secure and any software can crash at any time due to some simple mistake in the code. So to avoid all simple and silly mistakes and making more safe and secure code, below are the rules with no specific order you need to follow without fail.<br />
<br />
<ul style="text-align: left;">
<li>Static Code Analysis(Static analyzer)</li>
<li>Test Driven Development(TDD)</li>
<li>Code Review</li>
<li>Pair Programming</li>
</ul>
<br />
<b>Static Code Analysis:</b> Basically what static analysis will do is, it just scans our code and find out possible errors. There could be some copy-paste errors, some human errors etc. All these can be identified by Static analyzers. Static analysis done by static analyzer which is simply another software which can scan our code and generate report with all errors and warnings.There are lot of open source and commercial static analyzers available on the web. <a href="http://www.viva64.com/en/examples/">Here </a>are some of simple errors which identified by static analyzer.<br />
<br />
<b>Test Driven Development(TDD):</b> In TDD, first instead of writing code for functionality, need to write all possible test cases for that functionality. After finishing all test cases, run those test cases once. All these test cases will fail as there is no code available for the functionality. Now start writing the code to pass all these test cases. Believe me, It helps a lot in eliminating most of the bugs in the initial stage. Writing test cases for existing code is difficult. So always start tests cases before writing actual functionality. It will take some extra time, but it helps a lot.<br />
<br />
<b>Code Review: </b>This is one of the traditional way of finding out silly mistake done by developers. Always make sure that your code is reviewed by some one. Some others reviewing your code doesn't mean that you are not good in coding, it eliminates if any mistakes and it boasts your confidence levels if there are no comments :-) So Always go for the code review and don't skip it.<br />
<br />
<b>Pair Programming:</b> This is another new way of coding. Most of the developers thinks that if they are alone, they can write code quickly and efficiently. Yes that is true. But occasionally do pair programming. If possible code with new developer and some times with senior developer. While doing pair programming, basically two developers are seeing that code and two brains are working right!!. In this case, if any mistakes done, another developer identifies it and s/he may give another better way of writing the same code.<br />
<br />
Till now I have not specified any secure programming techniques rite? If you follow above rules, you can easily eliminate lot of common security related issues. All these steps are not specific to any particular programming language. In whatever language you are going to write, always follow these steps. Nowadays we have lot of IDE's (like <b>XCode</b>, Eclipse) which are supporting inbuilt frameworks to support static analyzers and TDD.<br />
<br />
Enjoy Coding!!!<br />
Happy Coding!!!<br />
<br />
<br />
References:<br />
<a href="http://www.viva64.com/en/b/0391/#ID0EK6PO">Click</a></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-11017868146445146162016-09-12T16:22:00.004+05:302016-09-12T16:44:50.904+05:30What is Internet of Things (IoT)?<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Now a days Broadband Internet or <b>WiFi </b>is available almost everywhere. So connecting our smart phone to internet is becomes easy due to availability of internet. We can find at least three devices like laptop, smart phone, tab in home and which are connected to WiFi. How about connecting Fridge or washing machine to WiFi? Idea of inter connecting all these devices is <b>Internet of Things</b> or shortly <b>IoT</b>.<br />
<br />
IoT in simple way is connecting internet enabled devices such as smart phone, fridge, washing machine or car etc.. Its a relation between people - people, things-things and people - things. These days each person on average using two internet enabled devices like smartphone or smartwatch or laptop. I see in future it will increase to five or six. In future every person on average is going to use five or six devices which can be connected to WiFi. So using IoT, we can connect Smart phone, Laptop, tab, washing machine, fridge etc ..<br />
<br />
There is a little concern about security and privacy about our personal information as these interconnected devices will share lot of our personal Data. But still IoT is in beginning stage, there wont be any problem if IoT uses good security methodologies to protect personal data.<br />
<br />
References regarding <b>IoT</b>:<br />
<br />
https://en.wikipedia.org/wiki/Internet_of_things<br />
http://www.forbes.com/sites/jacobmorgan/2014/05/13/simple-explanation-internet-things-that-anyone-can-understand/<br />
https://www.theguardian.com/technology/2015/may/06/what-is-the-internet-of-things-google<br />
<br />
Happy Learning!!!</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com1tag:blogger.com,1999:blog-2374446609565222827.post-69702423787189509092016-08-29T10:30:00.000+05:302016-08-29T10:30:10.806+05:30Archive upload failed due to the issues listed below!!!<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvVBywJsvQXgvW8aS1LYIvKJPQeZkLzpFvxdfxu65d6G9VKl-56XCfvxX8v5Knez-kdsmORmGJUAJiJyCixhYRJuzbrVuLYrWG8mUV-nL2qxlCmIt2ttLIZ14Lw03NxwJZaD7S0D2yxsEd/s1600/archive_upload_failed_due_to_the_issues_ios_ipa.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvVBywJsvQXgvW8aS1LYIvKJPQeZkLzpFvxdfxu65d6G9VKl-56XCfvxX8v5Knez-kdsmORmGJUAJiJyCixhYRJuzbrVuLYrWG8mUV-nL2qxlCmIt2ttLIZ14Lw03NxwJZaD7S0D2yxsEd/s400/archive_upload_failed_due_to_the_issues_ios_ipa.png" width="400" /></a></div>
<br />
<br />
<br />
When I tried to create a <b>archive</b> in <b>Xcode 7.3</b> and to upload <b>IPA</b> for <b>TestFlight</b>. I got this error. I tried it multiple times and got the same error. I googled for it and got simple solution and it worked!!. And the <span style="color: blue;">solution is trying it after some time</span>. <b>Yes</b> doing it after some time worked for me.<br />
<br />
This may be due to network issue or Apple server issue or some thing else. But It works if you try after some time. </div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com1tag:blogger.com,1999:blog-2374446609565222827.post-33138629261124934702016-08-25T12:08:00.002+05:302016-08-25T12:08:32.628+05:30Duplicate message reading from SQS - AWS !!<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Recently we faced one issue in reading messages from <b>SQS</b> in <b>AWS</b> cloud where we are processing same message multiple times. This issue we identified by using messsage identifier(<b>mid</b>) of the each message in AWS <b>REDSHIFT </b>table column.<br />
<br />
<b>How our system works:</b><br />
<br />
<ol style="text-align: left;">
<li>Post messages to AWS SQS using one task</li>
<li>Read batch of messages from SQS and start processing json message</li>
<li>Validate each message in a batch and put in AWS <b>S3 bucket</b></li>
<li>Load into AWS REDSHIFT database</li>
<li>Delete batch of messages from SQS after succesfuly processing messages</li>
</ol>
<br />
As I mentioned earlier we faced an issue of some of the messages processing multiple times and this was happening in <span style="color: red;">only one environment</span> and not all environments. To find RCA for this, it took almost three days and below is the RCA. Possible reasons for this to occur is,<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span><b>Reading a message from SQS and not deleting </b>- if this is the case messages never deletes from SQS and all messages should process multiple times. But this is not happening. Messages are deleting but some messages are processing multiple times<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span><b>Reading same message by multiple tasks to process </b>- This is happening in our scenario.<br />
Task1 reading batch of messages from SQS and before deleting all theses messages some other task picking some messages from task1 read messages. This is due to <b>visibility time out</b> of message in the SQS. Lets see how this happens.<br />
<br />
When a task read batch of messages from SQS , these messages are moved to <b><a href="http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/MonitorSQSwithCloudWatch.html">inflight mode</a> </b>and not visible to other task to read as these messages are under processing. And SQS has a property called visibility time out, so message in the <b>inflight</b> mode message are not visible to other tasks until this time out completes.<br />
<br />
Before this time out expires, we need to complete our process of message validation, loading to S3, storing to REDSHIFT and deleting. In our case some of the messages are not completing this process(or not deleting) with in the visibility time out(10sec in this case). So causing the message visible again after the visibility timeout expires.<br />
<br />
As I mentioned this was happening only one environment because, in that environment only we have visibility time out 10secs and all other enviroments we have 20secs. Our task is not completing with in 10secs and causing the duplicate message processing issue. To solve this issue, we just increased visibility time out to 20secs.<br />
<br />
I hope this helps.<br />
<br />
Happy Reading!!!<br />
<br />
<br />
<br />
<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-13782928831464741762016-03-13T07:45:00.001+05:302016-03-13T07:45:54.049+05:30Some of use full Programming Books!!<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div>
Below are some of use full programming books and sites. </div>
<div>
<div>
<b>Programming books:</b></div>
<div>
<a href="http://chimera.labs.oreilly.com/books/1234000001813/index.html" style="color: #111111;">Apprenticeship Pattern</a><br />
<div>
<a href="http://addyosmani.com/resources/essentialjsdesignpatterns/book/" style="color: #111111;">Learning JavaScript Design Patterns</a></div>
<div>
<a href="http://www.computingbook.org/" style="color: #111111;">Introduction to Computing</a></div>
<div>
<a href="https://leanpub.com/97-Things-Every-Programmer-Should-Know-Extended" style="color: #111111;">97 Things Every Programmer Should Know</a></div>
<div>
<a href="http://thecodelesscode.com/contents" style="color: #111111;">The Codeless Code</a></div>
<div>
<a href="http://htdp.org/" style="color: #111111;">How to Design Programs</a></div>
<div>
<a href="http://infolab.stanford.edu/~ullman/focs.html" style="color: #111111;">Foundations of Computer Science</a></div>
<div>
<a href="http://producingoss.com/" style="color: #111111;">Producing Open Source Software</a></div>
<div>
<a href="http://openmymind.net/FoundationsOfProgramming.pdf" style="color: #111111;">Foundations of Programming</a></div>
<div>
<a href="http://natureofcode.com/book/" style="color: #111111;">The Nature of Code</a></div>
<div>
<a href="http://dreamsongs.com/Files/PatternsOfSoftware.pdf" style="color: #111111;">Patterns of Software</a></div>
<div>
<a href="http://www.aosabook.org/en/index.html" style="color: #111111;">The Architecture of Open Source Applications</a></div>
<div>
<a href="http://openbookproject.net/thinkcs/python/english3e/" style="color: #111111;">How to Think Like a Computer Scientist</a></div>
<div>
<a href="http://regex.learncodethehardway.org/book/" style="color: #111111;">Learn Regex The Hard Way</a></div>
<div>
<a href="http://codingintro.com/" style="color: #111111;">The Little Introduction To Programming</a></div>
<div>
<a href="http://www.itmaybeahack.com/homepage/books/nonprogrammer.html" style="color: #111111;">Building Skills in Programming</a><br />
<a href="https://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwiKqMa1yrzLAhVRBY4KHfSVBloQFggcMAA&url=http%3A%2F%2Fvitoex.googlecode.com%2Fsvn%2Ftrunk%2FRead%2FClean%2520Code.pdf&usg=AFQjCNEQfyWOUS04JbSLCv2pPC9if5lt2w&sig2=RsjvXy4cfiFuyDlo11zENA&cad=rja">Uncle Bob Clean Code</a><br />
<br />
<br />
Happy Coding!!!<br />
<br />
<br />
Reference: <a href="http://www.techgig.com/tech-news/editors-pick/Free-Programming-Books-for-Coders-of-All-Levels-42954?mailer_id=2563&utm_source=Mailer&utm_medium=TG_batch&utm_campaign=digest_news_2016-03-12&etoken=Y2hhbmR1dGhlZGV2QGdtYWlsLmNvbQ==&activity_name=MTAxNTg4&template_type=3&dt=&auto_login=h6iCkYqamIulisObZKeOhI@@RUoavk55ISIBER1peXFt2V55ISIBER1dZV1t2XZJVVXk=&src_type=autoLogin">From Here</a></div>
</div>
</div>
</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-6986647649611285032016-02-20T19:43:00.001+05:302016-02-20T19:43:28.711+05:30S3 load errors in Redshift(AWS) COPY command<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<br />
We have faced lot of weird issues while loading <b>S3 bucke</b>t files into redshift. I will try to explain all issues what we faced. Before going that , lets see what are Valid S3 file should contain<br />
<br />
<ul style="text-align: left;">
<li>No.of values in S3 bucket are exactly equal to the no.of columns in the redshift table</li>
<li>Each value in S3 separated with a delimiter, in our case its pipe(|)</li>
<li>Each line in S3 file is exactly one insert statement on redshift</li>
<li>Empty values will be passed in the S3 file for corresponding optional field in table</li>
</ul>
<br />
<br />
To store S3 file content to redshift database, <b>AWS</b> provides a <b>COPY</b> command which stores bulk or batch of S3 data into redshift.<br />
Lets assume there is a table <i><span style="color: purple;"><b>testMessage</b></span></i> in redshift which has three columns <b><i><span style="color: purple;">id</span></i></b> of <i>integer</i> type, <b><i><span style="color: purple;">name</span></i></b> of <i>varchar</i>(10) type and <b><i><span style="color: purple;">msg</span></i></b> of <i>varchar</i>(10) type.<br />
<br />
S3 file to redshift inserting <b>COPY</b> command is below<br />
<br />
<b><i><span style="color: blue;">copy </span><span style="color: purple;">testMessage</span><span style="color: blue;"> (</span><span style="color: purple;">id, name, msg)</span><span style="color: blue;"> from '</span><span style="color: purple;">s3://blogpost.testbucket/test/file.txt</span><span style="color: blue;">' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_'</span></i></b><br />
<br />
To insert values from S3 file, sammple S3 file could be<br />
<br />
<span style="color: #3d85c6;">77</span><span style="color: red;">|</span><span style="color: #3d85c6;">chandu</span><span style="color: red;">|</span><span style="color: #3d85c6;">chanduthedev</span><br />
<br />
<br />
In this file total values are three which is equal to no.of columns in the testMessage table columns and each value separated by pipe(|) symbol.<br />
<br />
Lets see another S3 sample file<br />
<span style="color: #3d85c6;">88</span><span style="color: red;">|</span><span style="color: red;">|</span><span style="color: #3d85c6;">chanduthedev</span><br />
<br />
In this file, we have one empty value for name column in table testMessage in redshift. So far so good. Lets take some S3 files which cause to fail redshift COPY command<br />
<br />
<span style="color: #3d85c6;">99</span><span style="color: red;">|</span><span style="color: #3d85c6;">chanduthedev</span><br />
<br />
In this S3 file contains only two values 99 and chanduthdev, and missing third value which causes to file S3 load COPY command<br />
<br />
<span style="color: #3d85c6;">99</span><span style="color: red;">|</span><span style="color: #3d85c6;">ch</span><br />
<span style="color: #3d85c6;">and u</span><span style="color: red;">|</span><span style="color: #3d85c6;">chanduthedev</span><br />
<br />
In this file, second value is ch<span style="color: red;">\n</span>and u which conatins new line(\n) characters, so it becomes two rows in the S3 file which means two insert statements to redshift COPY command and. First row becomes two value insert statments which is invalid and second one is another two value invalid statement.<br />
<br />
For these invalid S3 file you may get below error message.<br />
<br />
<span style="color: red;">Load into table 'testMessage' failed. Check 'stl_load_errors' system table for details</span><br />
and in AWS you may get below error<br />
<span style="color: red;">Delimiter not found </span><br />
<br />
Lets take another failure S3 file which has delimiter as value for <i><span style="color: purple;"><b>name</b></span></i> column<br />
<br />
<span style="color: #3d85c6;">77</span><span style="color: red;">|</span><span style="color: #3d85c6;">chan</span><b>|</b><span style="color: #3d85c6;">234</span><span style="color: red;">|</span><span style="color: #3d85c6;">chanduthedev</span><br />
<span style="color: #3d85c6;"><br /></span>
<br />
In the above S3 file, it looks 4 values because of extra pipe(|) character for the <b><span style="color: purple;">name</span></b> chan|1234 which causes redshift <b>COPY</b> command to treat S3 file has four values, but table has three values.<br />
<br />
For S3 load failures, the most common reason could be <b>special characters</b> or <b>escape characters</b> like new line(\n), double quotes("), single quotes etc. Either you need to escape those special characters or remove those special characters.<br />
<br />
We followed later idea of removing special charasters while processing and storing in the redshift. But later came to know that we can use <b>ESCAPE</b> key word in COPY command.<br />
<br />
<br />
<b><i><span style="color: blue;">copy </span><span style="color: purple;">testMessage</span><span style="color: blue;"> (</span><span style="color: purple;">id, name, msg)</span><span style="color: blue;"> from '</span><span style="color: purple;">s3://blogpost.testbucket/test/file.txt</span><span style="color: blue;">' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE</span></i></b><br />
<br />
adding ESCAPE to COPY command will solve lot of these issues. So always check for ESCPAPE<br />
<br />
Happy Debugging...<br />
<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-62421833903323500992016-02-19T11:56:00.000+05:302016-02-19T11:56:59.955+05:30String length exceeds DDL length - S3 bucket Load error on redshift(AWS)<div dir="ltr" style="text-align: left;" trbidi="on">
We have recently faced one tricky issue in <b>AWS</b> <b>cloud</b> while loading <b>S3</b> file into <b>Redshift</b> using python. It took almost whole day to indentify the issue and fixing it.<br />
<br />
Our way of doing things in AWS cloud as below<br />
<ol style="text-align: left;">
<li>Get the json message from <b>SQS</b> using python</li>
<li>Validating the fields from json message received in step 1</li>
<li>Make a csv format after validating and store file in S3 bucket in AWS using csv python library</li>
<li>Load S3 file into AWS redshift database using copy command</li>
</ol>
<br />
The above process in simple terms, read the message, proces it and insert into redshift Database. In this process there could be a chance of failures like<br />
<ol style="text-align: left;">
<li>While validating json message from SQS, we may get invalid input which python cant identify like escape characters - <span style="color: magenta;">this also we faced and I will make another blog post on this soon</span></li>
<li>While write to s3 file we may get some extra escape characters - This problem we faced and I am covering now</li>
<li>While inserting into redshift db </li>
<ol>
<li>if we try to insert invalid datatype value in the column (for integer column trying to insert varchar value)</li>
<li>if the inserting value exceeds the length of the column (like for msg column lenght is 10, if we try to isnert more than 10 chars it will fail) - this was due to step 2</li>
<li>if S3 file does not have proper delimiters</li>
</ol>
</ol>
For simplifying big problem, I am assuming there is one table <b><span style="color: orange;">testMessage</span></b> in redshift which has two columns <span style="color: orange;">id</span> of <i><span style="color: blue;">integer</span></i> type and <span style="color: orange;">msg</span> of <span style="color: blue;"><i>varchar</i>(10)</span> type.<br />
To insert the values into <b><span style="color: orange;">testMessage</span></b> table using above process, we are expecting a json message which contains <span style="color: orange;">id</span> and <span style="color: orange;">msg</span> keys. Sample message shown below<br />
<br />
{"id":7, "msg":"testfile"}<br />
<br />
As per the above four step process<br />
<ol style="text-align: left;">
<li>json message from SQS</li>
<ul>
<li>{"id":7, "msg":"testfile"}</li>
</ul>
<li>Validating key - values in the json message</li>
<ul>
<li>looks valid as <span style="color: orange;">id</span> field contains integer and <span style="color: orange;">msg</span> field contains string less than or equal to 10 chars</li>
</ul>
<li>CSV format S3 file</li>
<ul>
<li>id<span style="color: red;">|</span>testfile</li>
</ul>
<li>Loading to redshift</li>
<ul>
<li>copy testMessage (id, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE</li>
</ul>
</ol>
This will work fine as there are no validation failures and no sepecial characters in the message. Lets take another message which contains special characters like double quotes<br />
<br />
{"id":7, "msg":"\"testfile\""}<br />
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: -webkit-standard; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; margin: 0px; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<br />
<ol style="-webkit-text-stroke-width: 0px; font-family: -webkit-standard; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<li style="color: black;">json message sqs message</li>
<ul style="color: black;">
<li>{"id":7, "msg":"\"testfile\""}</li>
</ul>
<li style="color: black;">Validating key - values in the json message</li>
<ul style="color: black;">
<li>looks valid as <span style="color: orange;">id</span> field contains integer and <span style="color: orange;">msg</span> field contains string less than or equal to 10 chars</li>
</ul>
<li style="color: black;">CSV format s3 file</li>
<ul>
<li>id<span style="color: red;">|</span>""testfile""</li>
</ul>
<li style="color: black;">Loading to redshift</li>
<ul style="color: black;">
<li>copy testMessage (id, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE</li>
</ul>
</ol>
<br />
<span style="color: red;">Bhooom .... </span><br />
Loading to redshift <b>copy</b> command fails with error message <span style="color: red;"><b>'String length exceeds DDL length '. </b></span><br />
Here comes the actual problem we faced!!!!!<br />
<br />
If we observe in step3 S3 file, actual <span style="color: orange;">msg</span> value is '"testfile"' whose length is 10, but we can <span style="color: red;">find two extra double quotes</span> in the s3 file which causes <span style="color: red;">exceeding</span> the length of the string to 12 from 10.<br />
From where all these extra double quotes has come??<br />
Until validations and processing message everything looks fine, but while writing into s3 file with CSV format, CSV library in python adds extra escape characters which causes actual problem.<br />
<br />
In our case we have double quotes which is a special character, and csv library adds another double quote as escape character which increase length from 10 to 12 which causes the problem<br />
<br />
<span style="color: blue;">To avoid this problem</span>, we can use<br />
<br />
<b>csv.register_dialect(dialect, doublequote=False, escapechar='\\', quoting=csv.QUOTE_NONE)</b><br />
<br />
This means we are making doublequotes as false and treating escape characters are empty.<br />
<br />
<ol style="text-align: left;">
<li><b>escapechar='\\'</b> this specifies, no escape characters are there, treat all special characters are normal characters.</li>
<li><b>escapechar=''</b> this doesn't mean no escape characters, which means empty as escape character</li>
</ol>
<br />
to pass empty value as escape characters use point 1 and not <b><span style="color: red;">point 2</span></b>. If you use you will get below error which means its expecting a escape character, but you have not specified any value<br />
<pre class="lang-py prettyprint prettyprinted"><code><span style="color: red;"><span class="typ">
</span></span></code></pre>
<pre class="lang-py prettyprint prettyprinted"><code><span style="color: red;"><span class="typ">Error</span><span class="pun">:</span><span class="pln"> need to escape</span><span class="pun">,</span><span class="pln"> but no escapechar set</span></span></code></pre>
<br />
This fix looks simple in this example, but in real time scenario we have almost 40 columns in a table and the SQS json message was more than 7000(7K) characters with lot of special characters like newline(\n), carriage return(\r), double slashes(\\), double quotes(") etc .... We worked a lot to identify this issue and sharing here to help others.<br />
<br />
<br />
Hope this helps ...<br />
Happy Debugging ......<br />
<br />
<br />
<br />
<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-85630936747977995552015-12-10T18:14:00.000+05:302015-12-10T18:14:03.274+05:30JSON vs XML which one is better?<div dir="ltr" style="text-align: left;" trbidi="on">
<b>JSON</b><br />
- good for web services<br />
- human readable format<br />
- it uses javascript eval() function which may cause some security flaw as we can excecute the js object<br />
<br />
<b>XML</b><br />
- good for configurations<br />
- Using XPath path, we can directly access the element<br />
- Using XSLT template, we can easily convert from Xml to json, csv or any format<br />
<br />
<br />
You can go with either JSON or XML basing on your requirement. Many developers think that JSON object is a light weight and which is better than XML. But in my opinion both are same.<br />
<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-61168084159162166332015-08-19T20:17:00.000+05:302015-08-19T20:17:46.436+05:30str, unicode and basestring - Python<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I faced one issue on Ubuntu 13.10 in python 2.7. I would like to share it here. the issue is regarding <b>str</b> and <b>unicode</b> type of strings.<br />
<br />
The scenario is I am checking for the type of a input, as I am expecting either string or dictionary (<b>json</b> object). So to know whether the input is string or dictionary, I am using python <b>isinstance </b>and checking for <b>str</b> and <b>dict</b> types. It was working fine in my local machine and failing in our dev environment for <b>str</b> type. The only difference between my local setup and dev envidonrment is Ubuntu OS version. In my local setup it was <b>Ubuntu 13.10</b> and in Dev it is <b>Ubuntu 14.04</b>.<br />
<br />
After struggling lot of hours finally found the issue is with <b>isinstance</b> for string data type. In <b>Ubuntu 13.10</b> setup isinstance is returning success for string type and in <b>Ubuntu 14.04</b> it was not, because in 14.04 string data type becomes <b>unicode</b> and not <b>str</b>. I am just checking for <b>str </b>and not checking for <b>unicode. </b>To avoid this problem we can check for <b>basestring </b>it will work for str and unicode and OS version independent.<br />
<br />
In python <b>basestring</b> is the parent for <b>str</b> and <b>unicode</b>. And unicode and str are siblings to each other. Check the below same python code.<br />
<br />
<pre class="brush:python">str_ip = "checking for str type"
unicode_ip = u'checking for unicode type'
# checking for str type
if isinstance(str_ip, str):
print "string type is str"
# checking for unicode type
if isinstance(unicode_ip, unicode):
print "string type unicode str"
# checking string for base string
if isinstance(str_ip, basestring):
print "str_ip is a string"
# checking unicode for base string
if isinstance(unicode_ip, basestring):
print "unicode_ip is a string"
</pre>
<br />
Happpy Coding!!!
</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-63832120456780333382015-06-27T10:33:00.005+05:302015-06-27T11:09:56.760+05:30Attempt to mutate immutable object with appendString - ObjectiveC<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
"Attempt to mutate immutable object with appendString" is one of the most common error while doing <b>iOS</b> development. This will occur due to mutability of the NSString. Lets check in detail.<br />
As most of you know foundation framework has two types of strings<br />
<br />
<ul style="text-align: left;">
<li><b>NSString</b> (not modifiable)</li>
<li><b>NSMutableString</b> (can modify)</li>
</ul>
<br />
Basing on these strings, we can have below two scenarios.<br />
<br />
1. We can assign NSMutableString to NSString<br />
2. We <span style="color: red;">can't</span> assign NSString to NSMutableString<br />
<br />
Scenario #1 is perfectly valid, because NSMutableString is a subclass of NSString. But scenario #2 will crash the application and throws below error message.<br />
<br />
"<b>Attempt to mutate immutable object with appendString</b>"<br />
<br />
<br />
To avoid this problem , while doing #2, you need to make mutableCopy to make a mutable string and assign. Check below for sample code.<br />
<br />
<pre class="brush:cpp">
//Immutable string , cant modify
NSString *firstName = @"Pasumarthi";
//mutable string can modify
NSMutableString *nameString = [NSMutableString stringWithString:@"Chandu"];
// invalid - trying to make mutable string to non-mutable string
//nameString = firstName;
//invalid - we cant make normal string copy to the mutable.
//nameString = [firstName copy];
// valid as mutablestring is subclass of string
nameString = [firstName mutableCopy];
// valid as mutablestring is subclass of string
firstName = nameString;
//valid as nameString is mutable
[nameString appendString:@" Appu"];
</pre>
<br />
Happy Coding!!!
<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-58348368339050685522015-06-17T20:26:00.000+05:302015-06-17T20:26:41.733+05:30XCode short cuts!!<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<b><br /></b></div>
<div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
When we are working with Integrated Development Kit(IDE) like Xcode, Eclipse, PyCharm etc, Short cuts will help you a lot and increases productivity. So here are the some of the short cuts which will be very useful for the <b>Xcode</b>. When developing iOS application these <b>Xcode short cuts</b> will help you a lot</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<br /></div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
Xcode is organized into four blocks as below</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<br /></div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
1. <b>Navigation Area:</b> where all the project files will be displayed in hierarchal order </div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
2. <b>Editing Area:</b> Where we can add/edit our source code to the files and this will be always visible most of the time, and no need to hide this area.</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
3. <b>Debug Area: </b>Where we can check the logs/message and stack frames when using break points</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
4. <b>Utility Area: </b>where we can find the properties of the UI elements and widgets etc.</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<b><br /></b></div>
<div class="separator" style="clear: both; font-family: 'Lucida Grande'; font-size: 12px; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj08RcGhicm5v1fZ1Yq6BXlDebPp8iJCmQsDpMF2cDvCtQQBzJef9f5FDRhvmVe6cPyYpW2l1UvGKJVKQ9v0O6Fxdx86i__EHyIOGnUIVMtNGdvUSEp67LHyM2RIfc6PefwbpMfEmKbdktd/s1600/xcodeVIew.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="188" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj08RcGhicm5v1fZ1Yq6BXlDebPp8iJCmQsDpMF2cDvCtQQBzJef9f5FDRhvmVe6cPyYpW2l1UvGKJVKQ9v0O6Fxdx86i__EHyIOGnUIVMtNGdvUSEp67LHyM2RIfc6PefwbpMfEmKbdktd/s320/xcodeVIew.png" width="320" /></a></div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<b><br /></b></div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<b><br /></b>
<b>Special keys on Mac Laptop: </b>Short cuts are combination of multiple keys mostly special keys and characters and numbers.<br />
<b><br /></b>
<b>⌘ - Command, Cmd</b></div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<span style="-webkit-text-stroke-color: rgb(0, 0, 0);"><b>⌃ - Control, Ctrl</b></span></div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
</div>
<div style="-webkit-text-stroke-width: 0px; font-family: 'Lucida Grande'; font-size: 12px; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 0px;">
<span style="-webkit-text-stroke-color: rgb(0, 0, 0);"><b>⌥ - Alt, Option, Opt</b></span></div>
<div style="margin: 0px;">
<span style="-webkit-text-stroke-color: rgb(0, 0, 0);"><code><b>⇧ - Shift</b></code></span></div>
<div style="color: black; font-weight: normal; margin: 0px;">
<span style="-webkit-text-stroke-color: rgb(0, 0, 0);"><br /></span></div>
</div>
<b><span style="font-family: Lucida Grande;"><span style="font-size: 12px;">Most frequently used and very usefull Xcode short cuts:</span></span></b><br />
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<br /></div>
</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<span style="-webkit-text-stroke-color: rgb(0, 0, 0); color: red;">⌘+0 - Navigation area </span><br />
<span style="color: blue;"><span style="-webkit-text-stroke-color: rgb(0, 0, 0);">⌘+</span><b>⌥ +</b>0 - Utility area</span><span style="color: red;"> </span><br />
<span style="color: red;"><span style="-webkit-text-stroke-color: rgb(0, 0, 0);">⌘+</span><span style="font-family: monospace;">⇧+Y</span> - Debug area </span><br />
<span style="color: blue;"><span style="-webkit-text-stroke-color: rgb(0, 0, 0);">⌘+</span><span style="font-family: monospace;">⇧+o</span> - File search and opening</span><br />
<span style="color: red;"><b>⌃+</b>⌘+Up/Down Arrow - <span style="font-family: Times; font-size: small;">switching/changing .m and .h</span></span><br />
<span style="color: blue;"><b>⌃+</b>⌘+Left/Right Arrow - <span style="font-family: Times; font-size: small;">go forward/back ward</span></span><br />
<span style="color: red;"><span style="font-family: 'Lucida Grande';">⌃</span> + 6 - to find a method in a file</span><br />
<span style="color: blue;"><br /></span>
<br />
<b>Below are some more use full short cuts:</b><br />
<b><br /></b>
#+1 - Project Navigator</div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
#+2 - Symbol Navigator<br />
#+3 - Find Navigator<br />
#+4 - Issue Navigator<br />
#+5 - Unit Tests Navigator<br />
#+6 - Debug Navigator<br />
#+7 - Break Point Navigator<br />
#+8 - Report Navigator<br />
#+b - Building project<br />
#+r - Running the application<br />
<span style="font-family: monospace;">⇧+</span>⌘+k - cleaning the project<br />
<br />
<br />
<br />
Happy Xcoding!!!<br />
<br />
<br /></div>
<div style="font-family: 'Lucida Grande'; font-size: 12px;">
<br /></div>
</div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-15380120724782783972014-12-14T15:53:00.000+05:302014-12-14T15:53:31.647+05:30Mysql date format and string parsing!!<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Recently I worked on Mysql and need to work on date and time formats and faced one issue where I need to convert from long date format(e.g Fir Dec 12 00:00:00 IST 2014 ) to date format <b>yyyy-mm-dd</b> format. So for doing this, we need to parse the long date format for that I used <b>concat, str_to_date</b> and <b>substring </b>inbuilt <b>Mysql</b> methods.<br />
<br />
<b>substring in Mysql: </b>Substring function used to get the substring from given string. For that we need to specify the ranges of the substring. Below is the example<br />
<b><br /></b>
Syntax:<br />
<br />
<b><i>substring</i>(input string,substringStartingPos[,lenghtOfTheString])</b><br />
<b><br /></b>
<b>Example:</b><br />
<b><br /></b>
<b>select substring('</b>Hello substring<b>',4)</b> gives 'lo substring' as it starts from 4th position to end of the string<br />
<br />
<b>select substring('</b>Hello substring<b>',4,4)</b> gives 'lo s' as it starts from the 4th position and it takes 4 characters as length specified 4<br />
<br />
<b>select substring('</b>Hello substring<b>',-4) </b>gives 'ring' as we specified the range four characters rom back(minus specifies from back).<br />
<br />
Similarly I have taken the Day, Month and year from the long date pattern format as below.<br />
<br />
<b>select substring('</b>Fir Dec 12 00:00:00 IST 2014<b>',-4) </b>gives '2014' which is a year<br />
<b>select substring</b>('Fir Dec 12 00:00:00 IST 2014',4,4) gives 'Dec' which is a month<br />
<b>select substring</b>('Fir Dec 12 00:00:00 IST 2014',9,2) gives '12' which is a day<br />
<br />
<b>Concat in Mysql: </b>Concat function will combine more than one string into one string. Below is the syntax and example.<br />
<br />
Syntax<b>:</b><br />
<b><br /></b>
<b><i>concat</i>(str1,str2,str3,....)</b><br />
<b><br /></b>
Example<b>:</b><br />
<b><br /></b>
<b>select concat</b>('Hello',' ', 'World!!') gives 'Hello World!!' as we passed three string to concat function.<br />
<br />
Concat string function used to separate the strings with special characters as delimiters.<br />
<br />
<br />
Coming to our problem of converting string to date format now as we know concat and substring functions. For that we need to use <b>str_to_date </b>inbuilt Mysql function.<br />
<br />
<b>str_to_date in Mysql: </b>str_to_date function as name says, it converts string to date format. Its inverse of <b>date_format</b>() function which converts from date to string.<br />
<br />
<br />
Syntax:<br />
<br />
<b>str_to_date(string,dateformat)</b><br />
<b><br /></b>
<b><br /></b>
Example:<br />
<b>select str_to_date</b>('Dec 12,2014', '%M %e,%Y') gives '2014-12-12' as %M is for month, %e as day and %Y is for Year. There are many formats for the date, we can check <a href="http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_date-format">here</a> for more.<br />
<br />
And finally the resulted query for converting from long date format to yyyy-mm-dd is below.<br />
<br />
<i>select <b>STR_TO_DATE</b>(<b>concat</b>(<b>substring</b>('Fir Dec 12 00:00:00 IST 2014',4,7),',',<b>substring</b>('Fir Dec 12 00:00:00 IST 2014',-4)), '%M %e,%Y') as date</i><br />
<br />
Which will give the date as '2014-12-12' which is required format.<br />
<br />
<br />
Happy Coding!!<br />
<br />
<br /></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0tag:blogger.com,1999:blog-2374446609565222827.post-14025479573615233242014-11-06T17:23:00.000+05:302014-11-06T17:23:15.104+05:30MySQL installation failure in OS X Yosemite!!<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I upgrade my MacBook Pro to latest Mac OS X version <b>Yosemite</b>. And when I try to install mysql, after completing the installation process , it is showing "<b>The installation failed</b>" message as shown below. I google for it and I found that actually mysql got installed successfully, but it was showing that message.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK4UGkWP980k5EuEFl9QCrzlg9ZpiksVZrTEyvQ6J1hHetvPeaPPyoxbKP6zecHxm7rnSYUKGwy2tUetksjqGRQq3TT4OE8t6lllBwxkgVLLallpZHPQyF1elyto57dYUei70Nc8xU-bBX/s1600/mysql-fail-osx-yosemite.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK4UGkWP980k5EuEFl9QCrzlg9ZpiksVZrTEyvQ6J1hHetvPeaPPyoxbKP6zecHxm7rnSYUKGwy2tUetksjqGRQq3TT4OE8t6lllBwxkgVLLallpZHPQyF1elyto57dYUei70Nc8xU-bBX/s1600/mysql-fail-osx-yosemite.png" height="222" width="320" /></a></div>
<br />
You can start working only mysql by starting mysql server. And below are the steps to do it.<br />
<br />
<b>To start mysql server:</b><br />
<br />
<i>sudo /usr/local/mysql/support-files/mysql.server start</i><br />
<b><br /></b>
<b>To stop mysql server:</b><br />
<i>sudo /usr/local/mysql/support-files/mysql.server stop</i><br />
<br />
<b>to login mysql via terminal:</b><br />
<i>/usr/local/mysql/bin/mysql</i><br />
<i><br /></i></div>
Chanduthedev phttp://www.blogger.com/profile/07353449039569824158noreply@blogger.com0