靠！sqoop2的文档太少了，而且居然不支持Hbase，十分简陋，所以我愤而放弃Sqoop2转为使用Sqoop1，之前跟着我教程看到朋友不要拿砖砸我，我是也是不知情的群众 卸载sqoop2 这步可选，如果你们是照着我之前的教程你们已经装了sqoop2就得先卸载掉，没装的可以跳
靠！sqoop2的文档太少了，而且居然不支持Hbase，十分简陋，所以我愤而放弃Sqoop2转为使用Sqoop1，之前跟着我教程看到朋友不要拿砖砸我，我是也是不知情的群众
卸载sqoop2
这步可选，如果你们是照着我之前的教程你们已经装了sqoop2就得先卸载掉，没装的可以跳过这步
$ sudo su - $ service sqoop2-server stop $ yum -y remove sqoop2-server $ yum -y remove sqoop2-client
安装Sqoop1
yum install -y sqoop
用help测试下是否有安装好
# sqoop help Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 14/11/28 11:33:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1 usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS job Work with saved jobs list-databases List available databases on a server list-tables List available tables in a database merge Merge results of incremental imports metastore Run a standalone Sqoop metastore version Display version information See 'sqoop help COMMAND' for information on a specific command.
拷贝驱动到 /usr/lib/sqoop/lib
mysql jdbc 驱动下载地址
下载后，解压开找到驱动jar包，upload到服务器上，然后移过去
mv /home/alex/mysql-connector-java-5.1.34-bin.jar /usr/lib/sqoop/lib
导入
数据准备
在mysql里面建立一个表CREATE TABLE `employee` ( `id` int(11) NOT NULL, `name` varchar(20) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
插入几条数据
insert into employee (id,name) values (1,'michael'); insert into employee (id,name) values (2,'ted'); insert into employee (id,name) values (3,'jack');
导入mysql到hdfs
列出所有表
我们先不急着导入，先做几个准备步骤热身一下，也方便排查问题列出所有数据库
# sqoop list-databases --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 14/12/01 09:20:28 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1 14/12/01 09:20:28 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 14/12/01 09:20:28 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. information_schema cacti metastore mysql sqoop_test wordpress zabbix
先用sqoop连接上数据库并列出所有表
# sqoop list-tables --connect jdbc:mysql://localhost/sqoop_test --username root --password root Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 14/11/28 11:46:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1 14/11/28 11:46:11 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 14/11/28 11:46:11 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. employee student workers
这条命令不用跟驱动的类名是因为sqoop默认支持mysql的，如果要跟jdbc驱动的类名用
# sqoop list-tables --connect jdbc:mysql://localhost/sqoop_test --username root --password root --driver com.mysql.jdbc.Driver
导入数据到hdfs
sqoop import --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root --table employee --m 1 --target-dir /user/test3
# sqoop import --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root --table employee --m 1 --target-dir /user/test Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 14/12/01 14:15:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1 14/12/01 14:15:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 14/12/01 14:15:41 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 14/12/01 14:15:41 INFO tool.CodeGenTool: Beginning code generation 14/12/01 14:15:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1 14/12/01 14:15:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1 14/12/01 14:15:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce Note: /tmp/sqoop-root/compile/7b8091924ce8deb4f2ccae14c404a5bf/employee.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 14/12/01 14:15:45 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/7b8091924ce8deb4f2ccae14c404a5bf/employee.jar 14/12/01 14:15:45 WARN manager.MySQLManager: It looks like you are importing from mysql. 14/12/01 14:15:45 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 14/12/01 14:15:45 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 14/12/01 14:15:45 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 14/12/01 14:15:45 INFO mapreduce.ImportJobBase: Beginning import of employee 14/12/01 14:15:46 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/12/01 14:15:47 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/12/01 14:15:47 INFO client.RMProxy: Connecting to ResourceManager at xmseapp01/10.172.78.111:8032 14/12/01 14:15:50 INFO db.DBInputFormat: Using read commited transaction isolation 14/12/01 14:15:51 INFO mapreduce.JobSubmitter: number of splits:1 14/12/01 14:15:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406097234796_0019 14/12/01 14:15:52 INFO impl.YarnClientImpl: Submitted application application_1406097234796_0019 14/12/01 14:15:52 INFO mapreduce.Job: The url to track the job: http://xmseapp01:8088/proxy/application_1406097234796_0019/ 14/12/01 14:15:52 INFO mapreduce.Job: Running job: job_1406097234796_0019 14/12/01 14:16:08 INFO mapreduce.Job: Job job_1406097234796_0019 running in uber mode : false 14/12/01 14:16:08 INFO mapreduce.Job: map 0% reduce 0% 14/12/01 14:16:19 INFO mapreduce.Job: map 100% reduce 0% 14/12/01 14:16:20 INFO mapreduce.Job: Job job_1406097234796_0019 completed successfully 14/12/01 14:16:21 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=99855 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=16 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=8714 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=8714 Total vcore-seconds taken by all map tasks=8714 Total megabyte-seconds taken by all map tasks=8923136 Map-Reduce Framework Map input records=2 Map output records=2 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=58 CPU time spent (ms)=1560 Physical memory (bytes) snapshot=183005184 Virtual memory (bytes) snapshot=704577536 Total committed heap usage (bytes)=148897792 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=16 14/12/01 14:16:21 INFO mapreduce.ImportJobBase: Transferred 16 bytes in 33.6243 seconds (0.4758 bytes/sec) 14/12/01 14:16:21 INFO mapreduce.ImportJobBase: Retrieved 2 records.
查看一下结果
# hdfs dfs -ls /user/test Found 2 items -rw-r--r-- 2 root supergroup 0 2014-12-01 14:16 /user/test/_SUCCESS -rw-r--r-- 2 root supergroup 16 2014-12-01 14:16 /user/test/part-m-00000 # hdfs dfs -cat /user/test/part-m-00000 1,michael 2,ted
我也不知道为什么mysql有3条数据，而导入了之后只有2条，有哪位懂的介绍下？
我遇到遇到的问题
如果你遇到以下问题14/12/01 10:12:42 INFO mapreduce.Job: Task Id : attempt_1406097234796_0017_m_000000_0, Status : FAILED Error: employee : Unsupported major.minor version 51.0用ps aux| grep hadoop看下会发现hadoop用的是jdk1.6 。我的cdh是5.0.1 sqoop版本是 1.4.4 ，我遇到了这个问题。
原因：sqoop是使用jdk1.7编译的，所以如果你用 ps aux| grep hadoop 看到hadoop用的是1.6运行的，那sqoop不能正常工作 注意：CDH4.7以上已经兼容jdk1.7 ，但如果你是从4.5升级上来的会发现hadoop用的是jdk1.6，需要修改一下整个hadoop调用的jdk为1.7，而且这是官方推荐的搭配
关于改jdk的方法
官方提供了2个方法 http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Requirements-and-Supported-Versions/cdhrsv_topic_3.html这个是让你把 /usr/java/ 下建一个软链叫 default 指向你要的jdk，我这么做了，无效 http://www.cloudera.com/content/cloudera/en/documentation/archives/cloudera-manager-4/v4-5-3/Cloudera-Manager-Enterprise-Edition-Installation-Guide/cmeeig_topic_16_2.html
这个是叫你增加一个环境变量， 我这么做了，无效 最后我用了简单粗暴的办法：停掉所有相关服务，然后删掉那个该死的jdk1.6然后再重启，这回就用了 /usr/java/default 了
停掉所有hadoop相关服务的命令
for x in `cd /etc/init.d ; ls hive-*` ; do sudo service $x stop ; done for x in `cd /etc/init.d ; ls hbase-*` ; do sudo service $x stop ; done /etc/init.d/zookeeper-server stop for x in `cd /etc/init.d ; ls hadoop-*` ; do sudo service $x stop ; done
zookeeper , hbase, hive 如果你们没装就跳过。建议你们用ps aux | grep jre1.6 去找找有什么服务，然后一个一个关掉，先关其他的，最后关hadoop
启动所有
for x in `cd /etc/init.d ; ls hadoop-*` ; do sudo service $x start ; done /etc/init.d/zookeeper-server start for x in `cd /etc/init.d ; ls hbase-*` ; do sudo service $x start ; done for x in `cd /etc/init.d ; ls hive-*` ; do sudo service $x start ; done
从hdfs导出数据到mysql
接着这个例子做数据准备
清空employeetruncate employee
导出数据到mysql
# sqoop export --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root --table employee --m 1 --export-dir /user/test Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 14/12/01 15:16:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1 14/12/01 15:16:50 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 14/12/01 15:16:51 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 14/12/01 15:16:51 INFO tool.CodeGenTool: Beginning code generation 14/12/01 15:16:51 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1 14/12/01 15:16:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1 14/12/01 15:16:52 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce Note: /tmp/sqoop-root/compile/f4a75fdefe1eb604181d47d6bc827e48/employee.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 14/12/01 15:16:55 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/f4a75fdefe1eb604181d47d6bc827e48/employee.jar 14/12/01 15:16:55 INFO mapreduce.ExportJobBase: Beginning export of employee 14/12/01 15:16:55 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/12/01 15:16:57 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 14/12/01 15:16:57 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/12/01 15:16:57 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/12/01 15:16:57 INFO client.RMProxy: Connecting to ResourceManager at xmseapp01/10.172.78.111:8032 14/12/01 15:17:00 INFO input.FileInputFormat: Total input paths to process : 1 14/12/01 15:17:00 INFO input.FileInputFormat: Total input paths to process : 1 14/12/01 15:17:00 INFO mapreduce.JobSubmitter: number of splits:1 14/12/01 15:17:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406097234796_0021 14/12/01 15:17:01 INFO impl.YarnClientImpl: Submitted application application_1406097234796_0021 14/12/01 15:17:01 INFO mapreduce.Job: The url to track the job: http://xmseapp01:8088/proxy/application_1406097234796_0021/ 14/12/01 15:17:01 INFO mapreduce.Job: Running job: job_1406097234796_0021 14/12/01 15:17:13 INFO mapreduce.Job: Job job_1406097234796_0021 running in uber mode : false 14/12/01 15:17:13 INFO mapreduce.Job: map 0% reduce 0% 14/12/01 15:17:21 INFO mapreduce.Job: Task Id : attempt_1406097234796_0021_m_000000_0, Status : FAILED Error: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'sqoop_test' at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:624) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:744) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'sqoop_test' at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) at com.mysql.jdbc.Util.getInstance(Util.java:360) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870) at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76) at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95) at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77) ... 8 more 14/12/01 15:17:29 INFO mapreduce.Job: Task Id : attempt_1406097234796_0021_m_000000_1, Status : FAILED Error: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'sqoop_test' at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:624) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:744) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'sqoop_test' at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) at com.mysql.jdbc.Util.getInstance(Util.java:360) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870) at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76) at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95) at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77) ... 8 more 14/12/01 15:17:40 INFO mapreduce.Job: map 100% reduce 0% 14/12/01 15:17:41 INFO mapreduce.Job: Job job_1406097234796_0021 completed successfully 14/12/01 15:17:41 INFO mapreduce.Job: Counters: 32 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=99542 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=139 HDFS: Number of bytes written=0 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job Counters Failed map tasks=2 Launched map tasks=3 Other local map tasks=2 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=21200 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=21200 Total vcore-seconds taken by all map tasks=21200 Total megabyte-seconds taken by all map tasks=21708800 Map-Reduce Framework Map input records=2 Map output records=2 Input split bytes=120 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=86 CPU time spent (ms)=1330 Physical memory (bytes) snapshot=177094656 Virtual memory (bytes) snapshot=686768128 Total committed heap usage (bytes)=148897792 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0 14/12/01 15:17:41 INFO mapreduce.ExportJobBase: Transferred 139 bytes in 43.6687 seconds (3.1831 bytes/sec) 14/12/01 15:17:41 INFO mapreduce.ExportJobBase: Exported 2 records.
那一串异常我也不知道为什么会有？！反正最后去mysql看成功导出了2条数据
mysql> select * from employee; +----+---------+ | id | name | +----+---------+ | 1 | michael | | 2 | ted | +----+---------+ 2 rows in set (0.00 sec)
好，下课！
声明
本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系admin@php.cn
Java错误：Hadoop错误，如何处理和避免Jun 24, 2023 pm 01:06 PMJava错误：Hadoop错误，如何处理和避免当使用Hadoop处理大数据时，常常会遇到一些Java异常错误，这些错误可能会影响任务的执行，导致数据处理失败。本文将介绍一些常见的Hadoop错误，并提供处理和避免这些错误的方法。Java.lang.OutOfMemoryErrorOutOfMemoryError是Java虚拟机内存不足的错误。当Hadoop任
3分钟快速使用ChatGPT教程，用它帮我写简历，太牛了Apr 11, 2023 pm 07:40 PM已经火了很久了，身边的同事也用它来进行一些调研，资源检索，工作汇报等方面都有很大的的效率提升。很多人问ChatGPT会不会取代程序员？我的回答是：不会！ChatGPT并不是我们的敌人，相反的是，它是我们的好帮手。未来人和人的竞争，可能就会从原先的我懂得更多，我实操经验更丰富，变成了我比你更会用工具，我比你更懂得提问，我比你更会发挥机器人的最大特性，所以，为了不掉队，你还不准备体验下ChatGPT吗？快速体验面试官经常会问你的项目有啥重难点？很多人不会回答，直接看看ChatGPT怎么说，真的太牛了
2023年最流行的5个php开发框架视频教程推荐May 08, 2017 pm 04:26 PM如果想快速进行php web开发，选择一个好用的php开发框架至关重要，一个好的php开发框架可以让开发工作变得更加快捷、安全和有效。那2023年最流行的php开发框架有哪些呢？这些php开发框架排名如何？
菜鸟如何开启派送通知Feb 29, 2024 pm 07:40 PM很多朋友会在菜鸟裹裹软件里查看自己的快递状态，有些朋友表示想知道怎样去设置开启派送通知。下面为大家介绍一下操作方法，还不了解的朋友一起来看看。1.打开手机中的菜鸟裹裹APP进入后，在页面的右下角位置点击“我的”切换进入。2.在我的页面里右上方位置点击“设置”图标打开。3.接下来，在设置页面里有一个“消息通知”，找到后在它的上面点击进入。4.在包裹通知设置页面里找到“派送中”这一项，在它的后面点击对应的开关按钮去进行设置，按钮为蓝色时代表开启该功能。当快递状态变更为派送中时会对我们进行通知提醒。
PHP基础教程：从入门到精通Jun 18, 2023 am 09:43 AMPHP是一种广泛使用的开源服务器端脚本语言，它可以处理Web开发中所有的任务。PHP在网页开发中的应用广泛，尤其是在动态数据处理上表现优异，因此被众多开发者喜爱和使用。在本篇文章中，我们将一步步地讲解PHP基础知识，帮助初学者从入门到精通。一、基本语法PHP是一种解释性语言，其代码类似于HTML、CSS和JavaScript。每个PHP语句都以分号;结束，注
什么是OCO订单?Apr 25, 2023 am 11:26 AM二选一订单（OneCancelstheOther，简称OCO）可让您同时下达两个订单。它结合了限价单和限价止损单，但只能执行其中一个。换句话说，只要其中的限价单被部分或全部成交、止盈止损单被触发，另一个订单将自动取消。请注意，取消其中一个订单也会同时取消另一个订单。在币安交易平台进行交易时，您可以将二选一订单作为交易自动化的基本形式。这个功能可让您选择同时下达两个限价单，从而有助于止盈和最大程度减少潜在损失。如何使用二选一订单？登录您的币安帐户之后，请前往基本交易界面，找到下图所示的交易区域。点
在Beego中使用Hadoop和HBase进行大数据存储和查询Jun 22, 2023 am 10:21 AM随着大数据时代的到来，数据处理和存储变得越来越重要，如何高效地管理和分析大量的数据也成为企业面临的挑战。Hadoop和HBase作为Apache基金会的两个项目，为大数据存储和分析提供了一种解决方案。本文将介绍如何在Beego中使用Hadoop和HBase进行大数据存储和查询。一、Hadoop和HBase简介Hadoop是一个开源的分布式存储和计算系统，它可
如何使用PHP和Hadoop进行大数据处理Jun 19, 2023 pm 02:24 PM随着数据量的不断增大，传统的数据处理方式已经无法处理大数据时代带来的挑战。Hadoop是开源的分布式计算框架，它通过分布式存储和处理大量的数据，解决了单节点服务器在大数据处理中带来的性能瓶颈问题。PHP是一种脚本语言，广泛应用于Web开发，而且具有快速开发、易于维护等优点。本文将介绍如何使用PHP和Hadoop进行大数据处理。什么是HadoopHadoop是
热AI工具
Undresser.AI Undress
人工智能驱动的应用程序，用于创建逼真的裸体照片
AI Clothes Remover
用于从照片中去除衣服的在线人工智能工具。
Undress AI Tool
免费脱衣服图片
Clothoff.io
AI脱衣机
AI Hentai Generator
免费生成ai无尽的。
热门文章
R.E.P.O.能量晶体解释及其做什么（黄色晶体）
3 周前By尊渡假赌尊渡假赌尊渡假赌
R.E.P.O.最佳图形设置
3 周前By尊渡假赌尊渡假赌尊渡假赌
刺客信条阴影：贝壳谜语解决方案
1 周前ByDDD
R.E.P.O.如果您听不到任何人，如何修复音频
3 周前By尊渡假赌尊渡假赌尊渡假赌
WWE 2K25：如何解锁Myrise中的所有内容
3 周前By尊渡假赌尊渡假赌尊渡假赌
热工具
SublimeText3 Mac版
神级代码编辑软件(SublimeText3)
DVWA
Damn Vulnerable Web App (DVWA) 是一个PHP/MySQL的Web应用程序，非常容易受到攻击。它的主要目标是成为安全专业人员在合法环境中测试自己的技能和工具的辅助工具，帮助Web开发人员更好地理解保护Web应用程序的过程，并帮助教师/学生在课堂环境中教授/学习Web应用程序安全。DVWA的目标是通过简单直接的界面练习一些最常见的Web漏洞，难度各不相同。请注意，该软件中
PhpStorm Mac 版本
最新（2018.2.1 ）专业的PHP集成开发工具
Atom编辑器mac版下载
最流行的的开源编辑器
MinGW - 适用于 Windows 的极简 GNU
这个项目正在迁移到osdn.net/projects/mingw的过程中，你可以继续在那里关注我们。MinGW：GNU编译器集合（GCC）的本地Windows移植版本，可自由分发的导入库和用于构建本地Windows应用程序的头文件；包括对MSVC运行时的扩展，以支持C99功能。MinGW的所有软件都可以在64位Windows平台上运行。
