搜索
首页数据库mysql教程Alex的Hadoop菜鸟教程:第8课Sqoop1安装/导入/导出教程

靠!sqoop2的文档太少了,而且居然不支持Hbase,十分简陋,所以我愤而放弃Sqoop2转为使用Sqoop1,之前跟着我教程看到朋友不要拿砖砸我,我是也是不知情的群众 卸载sqoop2 这步可选,如果你们是照着我之前的教程你们已经装了sqoop2就得先卸载掉,没装的可以跳

靠!sqoop2的文档太少了,而且居然不支持Hbase,十分简陋,所以我愤而放弃Sqoop2转为使用Sqoop1,之前跟着我教程看到朋友不要拿砖砸我,我是也是不知情的群众

卸载sqoop2

这步可选,如果你们是照着我之前的教程你们已经装了sqoop2就得先卸载掉,没装的可以跳过这步

$ sudo su -
$ service sqoop2-server stop
$ yum -y remove sqoop2-server
$ yum -y remove sqoop2-client

安装Sqoop1


 yum install -y sqoop


用help测试下是否有安装好
# sqoop help
Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
14/11/28 11:33:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

拷贝驱动到 /usr/lib/sqoop/lib

mysql jdbc 驱动下载地址

下载后,解压开找到驱动jar包,upload到服务器上,然后移过去

mv /home/alex/mysql-connector-java-5.1.34-bin.jar /usr/lib/sqoop/lib


导入

数据准备

在mysql里面建立一个表
CREATE TABLE `employee` (    
  `id` int(11) NOT NULL,    
  `name` varchar(20) NOT NULL,    
  PRIMARY KEY (`id`)    
) ENGINE=MyISAM  DEFAULT CHARSET=utf8;  

插入几条数据
insert into employee (id,name) values (1,'michael');  
insert into employee (id,name) values (2,'ted');
insert into employee (id,name) values (3,'jack'); 

导入mysql到hdfs

列出所有表

我们先不急着导入,先做几个准备步骤热身一下,也方便排查问题
列出所有数据库
# sqoop list-databases --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root
Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
14/12/01 09:20:28 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1
14/12/01 09:20:28 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/12/01 09:20:28 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
cacti
metastore
mysql
sqoop_test
wordpress
zabbix



先用sqoop连接上数据库并列出所有表
# sqoop list-tables --connect jdbc:mysql://localhost/sqoop_test --username root --password root
Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
14/11/28 11:46:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1
14/11/28 11:46:11 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/11/28 11:46:11 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
employee
student
workers

这条命令不用跟驱动的类名是因为sqoop默认支持mysql的,如果要跟jdbc驱动的类名用
# sqoop list-tables --connect jdbc:mysql://localhost/sqoop_test --username root --password root --driver com.mysql.jdbc.Driver

导入数据到hdfs

sqoop import --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root --table employee --m 1 --target-dir /user/test3

# sqoop import --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root --table employee --m 1 --target-dir /user/test
Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
14/12/01 14:15:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1
14/12/01 14:15:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/12/01 14:15:41 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/12/01 14:15:41 INFO tool.CodeGenTool: Beginning code generation
14/12/01 14:15:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
14/12/01 14:15:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
14/12/01 14:15:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/7b8091924ce8deb4f2ccae14c404a5bf/employee.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/12/01 14:15:45 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/7b8091924ce8deb4f2ccae14c404a5bf/employee.jar
14/12/01 14:15:45 WARN manager.MySQLManager: It looks like you are importing from mysql.
14/12/01 14:15:45 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
14/12/01 14:15:45 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
14/12/01 14:15:45 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
14/12/01 14:15:45 INFO mapreduce.ImportJobBase: Beginning import of employee
14/12/01 14:15:46 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/12/01 14:15:47 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/12/01 14:15:47 INFO client.RMProxy: Connecting to ResourceManager at xmseapp01/10.172.78.111:8032
14/12/01 14:15:50 INFO db.DBInputFormat: Using read commited transaction isolation
14/12/01 14:15:51 INFO mapreduce.JobSubmitter: number of splits:1
14/12/01 14:15:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406097234796_0019
14/12/01 14:15:52 INFO impl.YarnClientImpl: Submitted application application_1406097234796_0019
14/12/01 14:15:52 INFO mapreduce.Job: The url to track the job: http://xmseapp01:8088/proxy/application_1406097234796_0019/
14/12/01 14:15:52 INFO mapreduce.Job: Running job: job_1406097234796_0019
14/12/01 14:16:08 INFO mapreduce.Job: Job job_1406097234796_0019 running in uber mode : false
14/12/01 14:16:08 INFO mapreduce.Job:  map 0% reduce 0%
14/12/01 14:16:19 INFO mapreduce.Job:  map 100% reduce 0%
14/12/01 14:16:20 INFO mapreduce.Job: Job job_1406097234796_0019 completed successfully
14/12/01 14:16:21 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=99855
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=87
		HDFS: Number of bytes written=16
		HDFS: Number of read operations=4
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Other local map tasks=1
		Total time spent by all maps in occupied slots (ms)=8714
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=8714
		Total vcore-seconds taken by all map tasks=8714
		Total megabyte-seconds taken by all map tasks=8923136
	Map-Reduce Framework
		Map input records=2
		Map output records=2
		Input split bytes=87
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=58
		CPU time spent (ms)=1560
		Physical memory (bytes) snapshot=183005184
		Virtual memory (bytes) snapshot=704577536
		Total committed heap usage (bytes)=148897792
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=16
14/12/01 14:16:21 INFO mapreduce.ImportJobBase: Transferred 16 bytes in 33.6243 seconds (0.4758 bytes/sec)
14/12/01 14:16:21 INFO mapreduce.ImportJobBase: Retrieved 2 records.

查看一下结果
# hdfs dfs -ls /user/test
Found 2 items
-rw-r--r--   2 root supergroup          0 2014-12-01 14:16 /user/test/_SUCCESS
-rw-r--r--   2 root supergroup         16 2014-12-01 14:16 /user/test/part-m-00000
# hdfs dfs -cat /user/test/part-m-00000
1,michael
2,ted

我也不知道为什么mysql有3条数据,而导入了之后只有2条,有哪位懂的介绍下?

我遇到遇到的问题

如果你遇到以下问题
14/12/01 10:12:42 INFO mapreduce.Job: Task Id : attempt_1406097234796_0017_m_000000_0, Status : FAILED
Error: employee : Unsupported major.minor version 51.0
用ps aux| grep hadoop看下会发现hadoop用的是jdk1.6 。我的cdh是5.0.1 sqoop版本是 1.4.4 ,我遇到了这个问题。
原因:sqoop是使用jdk1.7编译的,所以如果你用 ps aux| grep hadoop 看到hadoop用的是1.6运行的,那sqoop不能正常工作 注意:CDH4.7以上已经兼容jdk1.7 ,但如果你是从4.5升级上来的会发现hadoop用的是jdk1.6,需要修改一下整个hadoop调用的jdk为1.7,而且这是官方推荐的搭配

关于改jdk的方法

官方提供了2个方法 http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Requirements-and-Supported-Versions/cdhrsv_topic_3.html
这个是让你把 /usr/java/ 下建一个软链叫 default 指向你要的jdk,我这么做了,无效 http://www.cloudera.com/content/cloudera/en/documentation/archives/cloudera-manager-4/v4-5-3/Cloudera-Manager-Enterprise-Edition-Installation-Guide/cmeeig_topic_16_2.html
这个是叫你增加一个环境变量, 我这么做了,无效 最后我用了简单粗暴的办法:停掉所有相关服务,然后删掉那个该死的jdk1.6然后再重启,这回就用了 /usr/java/default 了
停掉所有hadoop相关服务的命令
for x in `cd /etc/init.d ; ls hive-*` ; do sudo service $x stop ; done
for x in `cd /etc/init.d ; ls hbase-*` ; do sudo service $x stop ; done
/etc/init.d/zookeeper-server stop
for x in `cd /etc/init.d ; ls hadoop-*` ; do sudo service $x stop ; done


zookeeper , hbase, hive 如果你们没装就跳过。建议你们用ps aux | grep jre1.6 去找找有什么服务,然后一个一个关掉,先关其他的,最后关hadoop
启动所有

for x in `cd /etc/init.d ; ls hadoop-*` ; do sudo service $x start ; done
/etc/init.d/zookeeper-server start
for x in `cd /etc/init.d ; ls hbase-*` ; do sudo service $x start ; done
for x in `cd /etc/init.d ; ls hive-*` ; do sudo service $x start ; done

从hdfs导出数据到mysql

接着这个例子做

数据准备

清空employee
truncate employee

导出数据到mysql

# sqoop export --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root --table employee --m 1 --export-dir /user/test
Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
14/12/01 15:16:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.1
14/12/01 15:16:50 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/12/01 15:16:51 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/12/01 15:16:51 INFO tool.CodeGenTool: Beginning code generation
14/12/01 15:16:51 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
14/12/01 15:16:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
14/12/01 15:16:52 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/f4a75fdefe1eb604181d47d6bc827e48/employee.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/12/01 15:16:55 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/f4a75fdefe1eb604181d47d6bc827e48/employee.jar
14/12/01 15:16:55 INFO mapreduce.ExportJobBase: Beginning export of employee
14/12/01 15:16:55 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/12/01 15:16:57 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/12/01 15:16:57 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/12/01 15:16:57 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/12/01 15:16:57 INFO client.RMProxy: Connecting to ResourceManager at xmseapp01/10.172.78.111:8032
14/12/01 15:17:00 INFO input.FileInputFormat: Total input paths to process : 1
14/12/01 15:17:00 INFO input.FileInputFormat: Total input paths to process : 1
14/12/01 15:17:00 INFO mapreduce.JobSubmitter: number of splits:1
14/12/01 15:17:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406097234796_0021
14/12/01 15:17:01 INFO impl.YarnClientImpl: Submitted application application_1406097234796_0021
14/12/01 15:17:01 INFO mapreduce.Job: The url to track the job: http://xmseapp01:8088/proxy/application_1406097234796_0021/
14/12/01 15:17:01 INFO mapreduce.Job: Running job: job_1406097234796_0021
14/12/01 15:17:13 INFO mapreduce.Job: Job job_1406097234796_0021 running in uber mode : false
14/12/01 15:17:13 INFO mapreduce.Job:  map 0% reduce 0%
14/12/01 15:17:21 INFO mapreduce.Job: Task Id : attempt_1406097234796_0021_m_000000_0, Status : FAILED
Error: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'sqoop_test'
	at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:624)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:744)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database &#39;sqoop_test&#39;
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
	at com.mysql.jdbc.Util.getInstance(Util.java:360)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870)
	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
	at java.sql.DriverManager.getConnection(DriverManager.java:571)
	at java.sql.DriverManager.getConnection(DriverManager.java:215)
	at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
	at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
	at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
	... 8 more

14/12/01 15:17:29 INFO mapreduce.Job: Task Id : attempt_1406097234796_0021_m_000000_1, Status : FAILED
Error: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database &#39;sqoop_test&#39;
	at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:624)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:744)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database &#39;sqoop_test&#39;
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
	at com.mysql.jdbc.Util.getInstance(Util.java:360)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870)
	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
	at java.sql.DriverManager.getConnection(DriverManager.java:571)
	at java.sql.DriverManager.getConnection(DriverManager.java:215)
	at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
	at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
	at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
	at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
	... 8 more

14/12/01 15:17:40 INFO mapreduce.Job:  map 100% reduce 0%
14/12/01 15:17:41 INFO mapreduce.Job: Job job_1406097234796_0021 completed successfully
14/12/01 15:17:41 INFO mapreduce.Job: Counters: 32
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=99542
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=139
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=4
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=0
	Job Counters 
		Failed map tasks=2
		Launched map tasks=3
		Other local map tasks=2
		Rack-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=21200
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=21200
		Total vcore-seconds taken by all map tasks=21200
		Total megabyte-seconds taken by all map tasks=21708800
	Map-Reduce Framework
		Map input records=2
		Map output records=2
		Input split bytes=120
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=86
		CPU time spent (ms)=1330
		Physical memory (bytes) snapshot=177094656
		Virtual memory (bytes) snapshot=686768128
		Total committed heap usage (bytes)=148897792
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=0
14/12/01 15:17:41 INFO mapreduce.ExportJobBase: Transferred 139 bytes in 43.6687 seconds (3.1831 bytes/sec)
14/12/01 15:17:41 INFO mapreduce.ExportJobBase: Exported 2 records.


那一串异常我也不知道为什么会有?!反正最后去mysql看成功导出了2条数据
mysql> select * from employee;
+----+---------+
| id | name    |
+----+---------+
|  1 | michael |
|  2 | ted     |
+----+---------+
2 rows in set (0.00 sec)

好,下课!

声明
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系admin@php.cn
Java错误:Hadoop错误,如何处理和避免Java错误:Hadoop错误,如何处理和避免Jun 24, 2023 pm 01:06 PM

Java错误:Hadoop错误,如何处理和避免当使用Hadoop处理大数据时,常常会遇到一些Java异常错误,这些错误可能会影响任务的执行,导致数据处理失败。本文将介绍一些常见的Hadoop错误,并提供处理和避免这些错误的方法。Java.lang.OutOfMemoryErrorOutOfMemoryError是Java虚拟机内存不足的错误。当Hadoop任

3分钟快速使用ChatGPT教程,用它帮我写简历,太牛了3分钟快速使用ChatGPT教程,用它帮我写简历,太牛了Apr 11, 2023 pm 07:40 PM

已经火了很久了,身边的同事也用它来进行一些调研,资源检索,工作汇报等方面都有很大的的效率提升。很多人问ChatGPT会不会取代程序员?我的回答是:不会!ChatGPT并不是我们的敌人,相反的是,它是我们的好帮手。未来人和人的竞争,可能就会从原先的我懂得更多,我实操经验更丰富,变成了我比你更会用工具,我比你更懂得提问,我比你更会发挥机器人的最大特性,所以,为了不掉队,你还不准备体验下ChatGPT吗?快速体验面试官经常会问你的项目有啥重难点?很多人不会回答,直接看看ChatGPT怎么说,真的太牛了

菜鸟如何开启派送通知菜鸟如何开启派送通知Feb 29, 2024 pm 07:40 PM

很多朋友会在菜鸟裹裹软件里查看自己的快递状态,有些朋友表示想知道怎样去设置开启派送通知。下面为大家介绍一下操作方法,还不了解的朋友一起来看看。1.打开手机中的菜鸟裹裹APP进入后,在页面的右下角位置点击“我的”切换进入。2.在我的页面里右上方位置点击“设置”图标打开。3.接下来,在设置页面里有一个“消息通知”,找到后在它的上面点击进入。4.在包裹通知设置页面里找到“派送中”这一项,在它的后面点击对应的开关按钮去进行设置,按钮为蓝色时代表开启该功能。当快递状态变更为派送中时会对我们进行通知提醒。

2023年最流行的5个php开发框架视频教程推荐2023年最流行的5个php开发框架视频教程推荐May 08, 2017 pm 04:26 PM

如果想快速进行php web开发,选择一个好用的php开发框架至关重要,一个好的php开发框架可以让开发工作变得更加快捷、安全和有效。那2023年最流行的php开发框架有哪些呢?这些php开发框架排名如何?

PHP基础教程:从入门到精通PHP基础教程:从入门到精通Jun 18, 2023 am 09:43 AM

PHP是一种广泛使用的开源服务器端脚本语言,它可以处理Web开发中所有的任务。PHP在网页开发中的应用广泛,尤其是在动态数据处理上表现优异,因此被众多开发者喜爱和使用。在本篇文章中,我们将一步步地讲解PHP基础知识,帮助初学者从入门到精通。一、基本语法PHP是一种解释性语言,其代码类似于HTML、CSS和JavaScript。每个PHP语句都以分号;结束,注

什么是OCO订单?什么是OCO订单?Apr 25, 2023 am 11:26 AM

二选一订单(OneCancelstheOther,简称OCO)可让您同时下达两个订单。它结合了限价单和限价止损单,但只能执行其中一个。换句话说,只要其中的限价单被部分或全部成交、止盈止损单被触发,另一个订单将自动取消。请注意,取消其中一个订单也会同时取消另一个订单。在币安交易平台进行交易时,您可以将二选一订单作为交易自动化的基本形式。这个功能可让您选择同时下达两个限价单,从而有助于止盈和最大程度减少潜在损失。如何使用二选一订单?登录您的币安帐户之后,请前往基本交易界面,找到下图所示的交易区域。点

在Beego中使用Hadoop和HBase进行大数据存储和查询在Beego中使用Hadoop和HBase进行大数据存储和查询Jun 22, 2023 am 10:21 AM

随着大数据时代的到来,数据处理和存储变得越来越重要,如何高效地管理和分析大量的数据也成为企业面临的挑战。Hadoop和HBase作为Apache基金会的两个项目,为大数据存储和分析提供了一种解决方案。本文将介绍如何在Beego中使用Hadoop和HBase进行大数据存储和查询。一、Hadoop和HBase简介Hadoop是一个开源的分布式存储和计算系统,它可

如何使用PHP和Hadoop进行大数据处理如何使用PHP和Hadoop进行大数据处理Jun 19, 2023 pm 02:24 PM

随着数据量的不断增大,传统的数据处理方式已经无法处理大数据时代带来的挑战。Hadoop是开源的分布式计算框架,它通过分布式存储和处理大量的数据,解决了单节点服务器在大数据处理中带来的性能瓶颈问题。PHP是一种脚本语言,广泛应用于Web开发,而且具有快速开发、易于维护等优点。本文将介绍如何使用PHP和Hadoop进行大数据处理。什么是HadoopHadoop是

See all articles

热AI工具

Undresser.AI Undress

Undresser.AI Undress

人工智能驱动的应用程序,用于创建逼真的裸体照片

AI Clothes Remover

AI Clothes Remover

用于从照片中去除衣服的在线人工智能工具。

Undress AI Tool

Undress AI Tool

免费脱衣服图片

Clothoff.io

Clothoff.io

AI脱衣机

AI Hentai Generator

AI Hentai Generator

免费生成ai无尽的。

热门文章

R.E.P.O.能量晶体解释及其做什么(黄色晶体)
2 周前By尊渡假赌尊渡假赌尊渡假赌
仓库:如何复兴队友
1 个月前By尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island冒险:如何获得巨型种子
4 周前By尊渡假赌尊渡假赌尊渡假赌

热工具

安全考试浏览器

安全考试浏览器

Safe Exam Browser是一个安全的浏览器环境,用于安全地进行在线考试。该软件将任何计算机变成一个安全的工作站。它控制对任何实用工具的访问,并防止学生使用未经授权的资源。

SublimeText3 Linux新版

SublimeText3 Linux新版

SublimeText3 Linux最新版

SublimeText3汉化版

SublimeText3汉化版

中文版,非常好用

记事本++7.3.1

记事本++7.3.1

好用且免费的代码编辑器

SublimeText3 Mac版

SublimeText3 Mac版

神级代码编辑软件(SublimeText3)