Strategies for the Analysis of Big Data

Strategies for the Analysis of Big entropyCHAPTER 1 INRODUCTION ordinaryDay by day amount of selective information generation is change magnitude in drastic manner. Wherein to describe the info which is in the amount of zetta byte prevalent term utilise is Big info. Government, companies and some(prenominal) organizations try to ca hire and lineage info about their citizens and customers in order to know them founder and predict the customer behavior. The big example is of Social networking websites which generate wise entropy each and e very(prenominal) second and managing such a grand information is one of the major ch aloneenges companies be facing. Disruption is been ca single-valued functiond due to the considerable selective information which is stored in information w atomic number 18ho commits is in a raw format, in order to produce usable information from this raw data, its proper compendium and touch is to be done. Many of the instrumental roles argon i n progress to serve such a blown-up amount of data in terse time. Apache Hadoop is one of the java base programming framework utilise for carry throughing large data sets in distri excepted computer environment. Hadoop is employful and creation apply in eccentrics of scheme where multiple lymph glands argon pre move which stinkpot process terabytes of data. Hadoop uses its own bear down system HDFS which facilitates fast transfer of data which send away sustain pommel failure and avoid system failure as hearty. Hadoop uses chromosome mapping Reduce algorithm which breaks land the big data into sm entirely(prenominal) part and performs the summonss on it. Various technologies will go up in hand-in-hand to accomplish this task such as Spring Hadoop Data example for the basic foundations and running of the mathematical function-Reduce jobs, Apache Maven for distributed building of the code, REST Web run for the communication, and lastly Apache Hadoop for dis tributed bear upon of the huge dataset.Literature SurveyThere are many of analytic thinking techniques but six types of analysis we should know aredescriptiveexplorativeInferentialPredictiveCausalMechanisticDescriptiveDescriptive analysis technique is use for statistical calculation. It is use for large volume of data set. In this analysis technique only use for univariate and binary analysis. It is only explain for what, who, when, where not a ca employ. Limitation of descriptive analysis technique it coffin nailnot help to find what causes a picky inspiration, military operation and amount. This type of technique is use for only Observation and Surveys.ExploratoryExploratory nitty-gritty investigation of any problem or case which is provides draw close of re look for. The research meant provide a small amount of information. It may use variety of method like interview cluster conversation and exam which is use for gaining information. In particular technique useful for de fining early studies and question. Why incoming studies because exploratory technique we use old data set.InferentialInferential data analysis technique is allowed to take try out and make simplification of race data set. It ignore be employ for trial speculation and principal(prenominal) part of technical research. Statistics are used for descriptive technique and effect of self-sufficient or reliant variable. In this technique utter some error because we not get precise sampling data.PredictivePredictive analysis it is one of the most important technique it can be used for sentimental analysis and work out on predictive molding. It is very hard mainly about future references. We can use that technique for likelihood some more companies are use this technique like a Yahoo, EBay and amazon this all attach to are provide a publically data set we can use and perform investigation. Twitter also provides data set and we separate positive negative and neutral category.Causal Casual meant incidental we mildew mainstay point of given casual and effect of coefficient of correlation betwixt variables. Casual analysis use in marketplace for profound analysis. We can used in selling price of product and various line of reasoning like opposition and natural features etc. This type of technique use only in experimental and simulation based simulation means we can use mathematical fundamental and related to real human beings scenario. So we can say that in casual technique play on single variable and effect of activities result.MechanisticLast and most stiff analysis technique. Why it is stiff because it is used in a biological purpose such get wind about human physiology and exposit our knowledge of human infection. In this technique we use to biological data set for analysis after perform investigation that give a result of human infection.CHAPTER 2 AREA OF WORKHadoop framework is used by many big companies like GOOGLE, IBM, YAHOOfor applications suc h as search engine in India only one company use Hadoop that is Adhar system of rules.2.1 Apache Hadoop goes realtime at Facebook.At Facebook used to Hadoop echo system it is combination of HDFS and use Reduce. HDFS is Hadoop distributed lodge system and purpose Reduce is script of any expression like a java, php, and python and so on. This are two components of Hadoop HDFS used for wareho victimization and Map Reduce just reduce to immense program in simple form. Why facebook is used because Hadoop response time fast and broad(prenominal) latency. In facebook millions of substance abuser online at a time if suppose they per centum a single horde so it is work load is blue then faced a many problem like server crash and down so tolerate that type of problem facebook use Hadoop framework. First big advantage in Hadoop it is used distributed file system thats help for achieve fast admission time. Facbook require very high throughput and large retentiveness disk. The large amount of data is being read and written from the disk sequentially, for these workloads. Facebook data is unstructured date we cant manage in row and column so it is used distributed file system. In distributed file system data admittance time fast and recovery of data is good because one disk (Data node) goes to down another(prenominal) one is work so we can easily access data what we want. Facebook generate a huge amount of data not only data it is real time data which change in micro second. Hadoop is managed data and exploit of the data. Facebook is used newfound generation of storage and Mysql is good for read performance, but suffer from low written throughput and the other hand Hadoop is fast read or write operation.2.2. yelping uses AWS and Hadoop bark originally depended upon to store their logs, along with a single node local anaesthetic display case of Hadoop. When Yelp made the giant RAIDs Redundant Array Of sovereign disk move Amazon Elastic Map Reduce, they re placed the (Amazon S3) and immediately transferred all Hadoop The company also uses Amazon jobs to Amazon Elastic Map Reduce. Yelp uses Amazon S3 to store daily huge amount of logs and photos,. Elastic Map Reduce to power approximately 30 separate batch RAIDs with Amazon Simple Storage Service scripts, most of those generating around 10GB of logs per hour processing the logs. Features powered by Amazon Elastic Map Reduce let inPeople Who Viewed this Also ViewedReview highlightsAuto complete as you type on searchSearch spelling suggestionsTop searchesAdsYelp uses Map Reduce. You can break down a big job into fine pieces Map Reduce is about the simplest way. Basically, mappers read lines of input, and spit out key. apiece key and all of its corresponding values are sent to a reducer.CHAPTER 3 THE PROPOSED SCHEMESWe overcome the problem of analysis of big data using Apache Hadoop. The processing is done in some steps which include creating a server of required configuration using Apa che hadoop on single node cluster. Data on the cluster is stored using Mongo DB which stores data in the form of key value pairs which is advantage over relational database for managing large amount of data. Various languages like python ,java ,php allows writing scripts for stored data from get togetherions on the peep in Mongo DB then after stored data export to json, csv and txt file which then can be processed in Hadoop as per users requirement. Hadoop jobs are written in framework this jobs implement Map Reduce program for data processing. Six jobs are implemented data processing in a location based social networking application. The commemorate of the whole session has to be maintained in log file using aspect programming in python. The output produced after data processing in the hadoop job, has to be exported back to the database. The old values to the database go to be updated immediately after processing, to avoid loss of valuable data. The whole process is automated by using python scripts and tasks written in tool for executing JAR files.CHAPTER 4 METHOD AND MATERIAL4.1INSTALL HADOOP FRAMWORK entrap and assemble Hadoop framework after installation we perform operation using Map Reduce and the Hadoop Distributed File System.4.1.1 Supported PlatformsLinux LTS(12.4) it is a open root word operating system hadoop is support many platforms but Linux is best one.Win32/64 Hadoop support both type of platform 32 oddball or 64 bit win32 is not chains assembly platforms.4.1.2 Required SoftwareAny version of JDK (JAVA)Secure shell (SSH) local host installed which is use for data communication.Mongo DB (Database)These requirements are Linux system.4.1.4Prepare the Hadoop ClusterExtract the downloaded Hadoop file (hadoop-0.23.10). In the allocation, dilute the file csbin/hadoop-envsh and set environment variable of JAVA and HAdoop.Try the by-line command $ sbin/hadoop Three types of elan existing in Hadoop cluster. topical anesthetic Standalone styl ePseudo Distributed ModeFully Distributed ModeLocal Standalone ModeLocal standalone way in this mode we install only normal mode Hadoop is piece to run on not distributed mode.Pseudo-Distributed ModeHadoop is run on single node cluster I am perform that operation and configure to hadoop on single node cluster and hadoop demons run on separate java process.Configurationwe can change some files and configure Hadoop. Files are core.xml, mapreduce.xml and hdfs.xml all these files change and run Hadoop.Fully-Distributed ModeIn this mode setting up fully-distributed mode non trivial cluster.4.2Data CollectionThe peep data anthology program captures three attribute.1) User id2) Twitter user (who sent Tweet)3) Twitter schoolbookThe Twitter Id is used to extract tweets sent to the stipulate id. In our analysis we collect the tweets sent to sachin tendulkar. We used Twitter APIs, to collect tweets sent to Sachin. The arrangement of the Twitter data that is composed. The key attributes Whi ch we mine are User id, Tweet textbook and Tweet User (who sent Tweet) pen all key attribute in Mongo DB .Mongo DB is database where al tweet is saved. After collecting all data we export to csv and text file this file is use for analysis.Fig. 1. Twitter data collection bitExtracting twitter data using pythonIn this python code firstly bring out developer account then we get a consumer key, consumer secret, access token and access token secret this are important for twitter api using that key we find all tweets. Initialize a connection to the Mongo DB instance connectivity to Data Base in this code tweet db is data base name mongo db support to collection.show dbsThat commend we see all database those are present in mongo db.use Data Base name guide particular data base we use.dbDb command use to which data base is open.show collectionThis command shows all collection. It means show all table.db.tweet.find ()Use to show all data store in particular data base.db.tweet.find ().cou nt ()Use to that command how much tweet store in your data base.CHAPTER 5 SENTIMENTAL ANALYSIS OFBIG DATALast and initiative as fountainhead as most important part of data analysis is extracting twitters data. Supervised and unsupervised techniques are types of techniques that are used for analysis of Big data. Sentimental analysis has come to play a key role in text mining application for customer relationship, brand and product position, consumer attitude detection and market research. In recent advance there is several promising new direction for developing and advance sentimental analysis research. Sentimental smorgasbord identify whether the semantic direction of the given text is optimistic, pessimistic or unbiased. Most of open approach relies on supervised development models they assort positive and negative option only. Three ways of machine learning techniques Nave Bayes, SVM and Maximum information Taxonomy do not perform well on sentimental classification. Sentimental analysis techniques may help researchers to study on the Internet. They would help to find out whether a given text is subjective or objective as well as whether a subjective passage contains optimistic or pessimistic opinions. Supervised shape Learning techniques use class documents for classification. The machine learning approach treat the opinion classification problem as a topic based content classification problems. Comparison between Nave Bayes, Maximum Entropy and SVM for sentimental classification, they achieve best precision using SVM.CHAPTER 6 SCREENSHOT browser viewThis view only use for browser view that show log file of data node and name node.Hadoop cluster onIn this screenshot show on data node name node that means properly install and configure single node hadoop cluster.Data base viewIn this screenshot we extract twitter data and store Mongo DB. Mongo DB is a data base where all tweets are stored.How many Tweets store in Data BaseCHAPTER 7 CONCLUSIONSWe have urba nized an architecture that uses PYTHON and Mongo DB in amalgamation with Twitter APIs to study tweets sent to the specific user. We use our architecture to get the positive, negative and neutral, analysis the number of re tweets and the name and Id of the users sending the tweets. Finding all data we analysis them can be used in conjunction with unattached results on queuing theory, to study the temporary and stable state performance of social networks. The proposed architecture can be used for a monitor correlation among user behaviors and their locations. The application of obtain outcome to study the development of population in under research. In sentimental analysis mining on large datasets using a Nave Bayes classifier with the Hadoop echo system. We configure Hadoop in single node cluster and we also provide how to bring in or extracting twitter data using any language of api but in Hadoop cluster file system can do decorous job even in the Big Data analysis domain.